Editing Markov decision process (section)

=== Constrained Markov decision processes ===
Constrained Markov decision processes (CMDPS) are extensions to Markov decision process (MDPs). There are three fundamental differences between MDPs and CMDPs.<ref>{{cite book|last=Altman|first=Eitan|title=Constrained Markov decision processes|volume=7|publisher=CRC Press|year=1999}}</ref>

* There are multiple costs incurred after applying an action instead of one.
* CMDPs are solved with [[Linear programming|linear programs]] only, and [[dynamic programming]] does not work.
* The final policy depends on the starting state.

The method of Lagrange multipliers applies to CMDPs.
Many Lagrangian-based algorithms have been developed.
* Natural policy gradient primal-dual method.<ref>
{{cite conference
 | last1 = Ding | first1 =Dongsheng
 | last2 = Zhang | first2 = Kaiqing
 | last3 = Jovanovic | first3 = Mihailo
 | last4 = Basar | first4 = Tamer
 | year = 2020
 | title = Natural policy gradient primal-dual method for constrained Markov decision processes
 | conference = Advances in Neural Information Processing Systems
}}
</ref>

There are a number of applications for CMDPs. It has recently been used in [[motion planning]] scenarios in robotics.<ref>{{cite conference |last1=Feyzabadi |first1=S. |last2=Carpin |first2=S. |title=Risk-aware path planning using hierarchical constrained Markov Decision Processes |book-title=Automation Science and Engineering (CASE) |conference=IEEE International Conference |pages=297, 303 |date=18–22 Aug 2014 |url=https://www.researchgate.net/publication/270105954}}</ref>