Editing Pontryagin's maximum principle

{{short description|Principle in optimal control theory for best way to change state in a dynamical system}}
'''Pontryagin's maximum principle''' is used in [[optimal control]] theory to find the best possible control for taking a [[dynamical system]] from one state to another, especially in the presence of constraints for the state or input controls. It states that it is [[Necessity and sufficiency|necessary]] for any optimal control along with the optimal state trajectory to solve the so-called Hamiltonian system, which is a two-point [[boundary value problem]], plus a maximum condition of the [[Hamiltonian (control theory)|control Hamiltonian]].{{efn|Whether the extreme value is maximum or minimum depends on the sign convention used for defining the Hamiltonian. The historic convention leads to a maximum, hence ''maximum principle.''  In recent years, it is more commonly referred to as simply Pontryagin's Principle, without the use of the adjectives, maximum or minimum.}} These necessary conditions become sufficient under certain [[Convex_function|convexity]] conditions on the objective and constraint functions.<ref>{{cite journal |journal=SIAM Journal on Control |volume=4 |year=1966 |issue=1 |pages=139–152 |title=Sufficient Conditions for the Optimal Control of Nonlinear Systems |first=O. L. |last=Mangasarian |author-link=Olvi L. Mangasarian |doi=10.1137/0304013 }}</ref><ref>{{cite journal |first1=Morton I. |last1=Kamien |author-link=Morton Kamien |first2=Nancy L. |last2=Schwartz |title=Sufficient Conditions in Optimal Control Theory |journal=[[Journal of Economic Theory]] |volume=3 |issue=2 |year=1971 |pages=207–214 |doi=10.1016/0022-0531(71)90018-4 }}</ref>

The maximum principle was formulated in 1956 by the Russian mathematician [[Lev Pontryagin]] and his students,<ref>{{cite book |first1=V. |last1=Boltyanski |first2=H. |last2=Martini |first3=V. |last3=Soltan |title=Geometric Methods and Optimization Problems |location=New York |publisher=Springer |year=1998 |isbn=0-7923-5454-0 |pages=204–227 |chapter=The Maximum Principle – How it came to be? |chapter-url=https://books.google.com/books?id=YD7UBwAAQBAJ&pg=PA204 }}</ref><ref>{{cite journal |first=R. V. |last=Gamkrelidze |title=Discovery of the Maximum Principle |journal=Journal of Dynamical and Control Systems |year=1999 |volume=5 |issue=4 |pages=437–451 |doi=10.1023/A:1021783020548 |s2cid=122690986 }} Reprinted in {{cite book |editor-first=A. A. |editor-last=Bolibruch |editor-link=Andrei Bolibrukh |editor2-first=Yu. S. |editor2-last=Osipov |editor3-first=Ya. G. |editor3-last=Sinai |display-editors=1 |title=Mathematical Events of the Twentieth Century |location=Berlin |publisher=Springer |year=2006 |isbn=3-540-23235-4 |pages=85–99 }}</ref> and its initial application was to the maximization of the terminal speed of a rocket.<ref>For first published works, see references in {{cite journal |last=Fuller |first=A. T. |title=Bibliography of Pontryagin's Maximum Principle |journal=J. Electronics & Control |volume=15 |issue=5 |year=1963 |pages=513–517 |doi=10.1080/00207216308937602 }}</ref> The result was derived using ideas from the classical [[calculus of variations]].<ref>{{cite journal |first=E. J. |last=McShane |author-link=Edward J. McShane |year=1989 |title=The Calculus of Variations from the Beginning Through Optimal Control Theory |journal=SIAM J. Control Optim. |volume=27 |issue=5 |pages=916–939 |doi=10.1137/0327049 }}</ref> After a slight [[Perturbation function|perturbation]] of the optimal control, one considers the first-order term of a [[Taylor Series|Taylor]] expansion with respect to the perturbation; sending the perturbation to zero leads to a variational inequality from which the maximum principle follows.<ref name="YongZhou">{{cite book |last1=Yong |first1=J. |last2=Zhou |first2=X. Y. |title=Stochastic Controls: Hamiltonian Systems and HJB Equations |url=https://archive.org/details/stochasticcontro00yong |url-access=limited |chapter=Maximum Principle and Stochastic Hamiltonian Systems |date=1999 |publisher=Springer |location=New York |isbn=0-387-98723-1 |pages=[https://archive.org/details/stochasticcontro00yong/page/n63 101]–156 }}</ref>

Widely regarded as a milestone in optimal control theory, the significance of the maximum principle lies in the fact that maximizing the Hamiltonian is much easier than the original infinite-dimensional control problem; rather than maximizing over a [[function space]], the problem is converted to a [[pointwise]] optimization.<ref>{{cite web |first=Shankar |last=Sastry |title=Lecture Notes 8. Optimal Control and Dynamic Games |date=March 29, 2009 |url=https://inst.eecs.berkeley.edu/~ee291e/sp09/handouts/09-lecture8-rev.pdf }}</ref> A similar logic leads to [[Bellman's principle of optimality]], a related approach to optimal control problems which states that the optimal trajectory remains optimal at intermediate points in time.<ref>{{cite journal |first=X. Y. |last=Zhou |title=Maximum Principle, Dynamic Programming, and their Connection in Deterministic Control |journal=Journal of Optimization Theory and Applications |year=1990 |volume=65 |issue=2 |pages=363–373 |doi=10.1007/BF01102352 |s2cid=122333807 }}</ref> The resulting [[Hamilton–Jacobi–Bellman equation]] provides a necessary and sufficient condition for an optimum, and admits [[Hamilton–Jacobi–Bellman equation#Extension to stochastic problems|a straightforward extension]] to stochastic optimal control problems, whereas the maximum principle does not.<ref name="YongZhou" />  However, in contrast to the Hamilton–Jacobi–Bellman equation, which needs to hold over the entire state space to be valid, Pontryagin's Maximum Principle is potentially more computationally efficient in that the conditions which it specifies only need to hold over a particular trajectory.

==Notation==
For set <math>\mathcal{U}</math> and functions 
:<math>\Psi : \reals^n \to \reals</math>,
:<math>H : \reals^n \times \mathcal{U} \times \reals^n \times \reals \to \reals</math>,
:<math>L : \reals^n \times \mathcal{U} \to \reals</math>,
:<math>f : \reals^n \times \mathcal{U} \to \reals^n</math>,
we use the following notation:
:<math>
\Psi_T(x(T))= \left.\frac{\partial \Psi(x)}{\partial T}\right|_{x=x(T)} \,
</math>,

:<math>
\Psi_x(x(T))=\begin{bmatrix} \left.\frac{\partial
\Psi(x)}{\partial x_1}\right|_{x=x(T)} & \cdots & \left.\frac{\partial
\Psi(x)}{\partial x_n} \right|_{x=x(T)}
\end{bmatrix}
</math>,

:<math>
H_x(x^*,u^*,\lambda^*,t)=\begin{bmatrix} \left.\frac{\partial H}{\partial x_1}\right|_{x=x^*,u=u^*,\lambda=\lambda^*}
& \cdots & \left.\frac{\partial H}{\partial x_n}\right|_{x=x^*,u=u^*,\lambda=\lambda^*}
\end{bmatrix}
</math>,

:<math>
L_x(x^*,u^*)=\begin{bmatrix} \left.\frac{\partial L}{\partial x_1}\right|_{x=x^*,u=u^*}
& \cdots & \left.\frac{\partial L}{\partial x_n}\right|_{x=x^*,u=u^*}
\end{bmatrix}
</math>,
:<math>
f_x(x^*,u^*)=\begin{bmatrix} \left.\frac{\partial f_1}{\partial x_1}\right|_{x=x^*,u=u^*} & \cdots & \left.\frac{\partial f_1}{\partial x_n}\right|_{x=x^*,u=u^*} \\
\vdots & \ddots & \vdots \\ \left.\frac{\partial f_n}{\partial x_1}\right|_{x=x^*,u=u^*} &
\ldots & \left.\frac{\partial f_n}{\partial x_n}\right|_{x=x^*,u=u^*}
\end{bmatrix}
</math>.

==Formal statement of necessary conditions for minimization problems==
Here the necessary conditions are shown for minimization of a functional. 

Consider an n-dimensional [[dynamical system]], with state variable <math>x \in \R^n</math>, and control variable <math>u \in \mathcal{U}</math>, where <math>\mathcal{U}</math> is the set of admissible controls. The evolution of the system is determined by the state and the control, according to the differential equation <math>\dot{x}=f(x,u)</math>. Let the system's initial state be <math>x_0</math> and let the system's evolution be controlled over the time-period with values <math>t \in [0, T]</math>. The latter is determined by the following differential equation:
:<math>
\dot{x}=f(x,u), \quad x(0)=x_0, \quad u(t) \in \mathcal{U}, \quad t \in [0,T]
</math>
The control trajectory <math>u: [0, T] \to \mathcal{U}</math> is to be chosen according to an objective. The objective is a functional <math>J</math> defined by
:<math>
J=\Psi(x(T))+\int^T_0 L\big(x(t),u(t)\big) \,dt
</math>,
where <math>L(x, u)</math> can be interpreted as the ''rate'' of cost for exerting control <math>u</math> in state <math>x</math>, and <math>\Psi(x)</math> can be interpreted as the cost for ending up at state <math>x</math>. The specific choice of <math>L, \Psi</math> depends on the application.

The constraints on the system dynamics can be adjoined to the [[Lagrangian mechanics|Lagrangian]] <math>L</math> by introducing time-varying [[Lagrange multipliers|Lagrange multiplier]] vector <math>\lambda</math>, whose elements are called the ''costates'' of the system. This motivates the construction of the [[Hamiltonian (control theory)|Hamiltonian]] <math>H</math> defined for all <math>t \in [0,T]</math> by:
:<math>
H\big(x(t),u(t),\lambda(t),t\big)=\lambda^{\rm T}(t)\cdot f\big(x(t),u(t)\big) + L\big(x(t),u(t)\big)
</math>
where <math>\lambda^{\rm T}</math> is the transpose of <math>\lambda</math>.

Pontryagin's minimum principle states that the optimal state trajectory <math>x^*</math>, optimal control <math>u^*</math>, and corresponding Lagrange multiplier vector <math>\lambda^*</math> must  minimize the Hamiltonian <math>H</math> so that

{{NumBlk|:|<math> H\big(x^*(t),u^*(t),\lambda^*(t),t\big)\leq H\big(x(t),u,\lambda(t),t\big) </math>|{{EquationRef|1}}}}

for all time <math>t \in [0,T]</math> and for all permissible control inputs <math>u \in \mathcal{U}</math>. Here, the trajectory of the Lagrangian multiplier vector <math>\lambda</math> is the solution to the [[costate equation]] and its terminal conditions:

{{NumBlk|:|<math> -\dot{\lambda}^{\rm T}(t)=H_x\big(x^*(t),u^*(t),\lambda(t),t\big)=\lambda^{\rm T}(t)\cdot f_x\big(x^*(t),u^*(t)\big) + L_x\big(x^*(t),u^*(t)\big)</math>|{{EquationRef|2}}}}

{{NumBlk|:|<math> \lambda^{\rm T}(T)=\Psi_x(x(T)) </math>|{{EquationRef|3}}}}

If <math>x(T)</math> is fixed, then these three conditions in (1)-(3) are the necessary conditions for an optimal control. 

If the final state <math>x(T)</math> is not fixed (i.e., its differential variation is not zero), there is an additional condition

{{NumBlk|:|<math> \Psi_T(x(T))+H(T)=0 </math>|{{EquationRef|4}}}}

These four conditions in (1)-(4) are the necessary conditions for an optimal control.

==See also==
* [[Lagrange multipliers on Banach spaces]], Lagrangian method in calculus of variations

==Notes==
{{notelist}}

==References==
{{Reflist}}

==Further reading==
* {{cite book |last=Geering |first=H. P. |title=Optimal Control with Engineering Applications |publisher=Springer |year=2007 |isbn=978-3-540-69437-3 }}
* {{cite book |last=Kirk |first=D. E. |title=Optimal Control Theory: An Introduction |publisher=Prentice Hall |year=1970 |isbn=0-486-43484-2 }}
* {{cite book |last1=Lee |first1=E. B. |first2=L. |last2=Markus |title=Foundations of Optimal Control Theory |url=https://archive.org/details/foundationsofopt0000leee |url-access=registration |location=New York |publisher=Wiley |year=1967 }}
* {{cite book |first1=Atle |last1=Seierstad |first2=Knut |last2=Sydsæter |author-link2=Knut Sydsæter |title=Optimal Control Theory with Economic Applications |location=Amsterdam |publisher=North-Holland |year=1987 |isbn=0-444-87923-4 }}

==External links==
* {{springer|title=Pontryagin maximum principle|id=Pontryagin_maximum_principle&oldid=38944}}

{{DEFAULTSORT:Pontryagin's Minimum Principle}}
[[Category:Optimal control]]
[[Category:Principles]]

[[fr:Commande optimale#Principe du maximum]]
[[ru:Оптимальное управление#Принцип максимума Понтрягина]]