arrow
Volume 42, Issue 3
Approximating the Stationary Bellman Equation by Hierarchical Tensor Products

Mathias Oster, Leon Sallandt & Reinhold Schneider

J. Comp. Math., 42 (2024), pp. 638-661.

Published online: 2024-04

Export citation
  • Abstract

We treat infinite horizon optimal control problems by solving the associated stationary Bellman equation numerically to compute the value function and an optimal feedback law. The dynamical systems under consideration are spatial discretizations of non linear parabolic partial differential equations (PDE), which means that the Bellman equation suffers from the curse of dimensionality. Its non linearity is handled by the Policy Iteration algorithm, where the problem is reduced to a sequence of linear equations, which remain the computational bottleneck due to their high dimensions. We reformulate the linearized Bellman equations via the Koopman operator into an operator equation, that is solved using a minimal residual method. Using the Koopman operator we identify a preconditioner for operator equation, which deems essential in our numerical tests. To overcome computational infeasibility we use low rank hierarchical tensor product approximation/tree-based tensor formats, in particular tensor trains (TT tensors) and multi-polynomials, together with high-dimensional quadrature, e.g. Monte-Carlo. By controlling a destabilized version of viscous Burgers and a diffusion equation with unstable reaction term numerical evidence is given.

  • AMS Subject Headings

49L20, 15A69, 49M41, 49N35

  • Copyright

COPYRIGHT: © Global Science Press

  • Email address
  • BibTex
  • RIS
  • TXT
@Article{JCM-42-638, author = {Oster , MathiasSallandt , Leon and Schneider , Reinhold}, title = {Approximating the Stationary Bellman Equation by Hierarchical Tensor Products}, journal = {Journal of Computational Mathematics}, year = {2024}, volume = {42}, number = {3}, pages = {638--661}, abstract = {

We treat infinite horizon optimal control problems by solving the associated stationary Bellman equation numerically to compute the value function and an optimal feedback law. The dynamical systems under consideration are spatial discretizations of non linear parabolic partial differential equations (PDE), which means that the Bellman equation suffers from the curse of dimensionality. Its non linearity is handled by the Policy Iteration algorithm, where the problem is reduced to a sequence of linear equations, which remain the computational bottleneck due to their high dimensions. We reformulate the linearized Bellman equations via the Koopman operator into an operator equation, that is solved using a minimal residual method. Using the Koopman operator we identify a preconditioner for operator equation, which deems essential in our numerical tests. To overcome computational infeasibility we use low rank hierarchical tensor product approximation/tree-based tensor formats, in particular tensor trains (TT tensors) and multi-polynomials, together with high-dimensional quadrature, e.g. Monte-Carlo. By controlling a destabilized version of viscous Burgers and a diffusion equation with unstable reaction term numerical evidence is given.

}, issn = {1991-7139}, doi = {https://doi.org/10.4208/jcm.2112-m2021-0084}, url = {http://global-sci.org/intro/article_detail/jcm/23030.html} }
TY - JOUR T1 - Approximating the Stationary Bellman Equation by Hierarchical Tensor Products AU - Oster , Mathias AU - Sallandt , Leon AU - Schneider , Reinhold JO - Journal of Computational Mathematics VL - 3 SP - 638 EP - 661 PY - 2024 DA - 2024/04 SN - 42 DO - http://doi.org/10.4208/jcm.2112-m2021-0084 UR - https://global-sci.org/intro/article_detail/jcm/23030.html KW - Feedback control, Dynamic programming, Hamilton-Jacobi-Bellman, Tensor product approximation, Variational Monte-Carlo. AB -

We treat infinite horizon optimal control problems by solving the associated stationary Bellman equation numerically to compute the value function and an optimal feedback law. The dynamical systems under consideration are spatial discretizations of non linear parabolic partial differential equations (PDE), which means that the Bellman equation suffers from the curse of dimensionality. Its non linearity is handled by the Policy Iteration algorithm, where the problem is reduced to a sequence of linear equations, which remain the computational bottleneck due to their high dimensions. We reformulate the linearized Bellman equations via the Koopman operator into an operator equation, that is solved using a minimal residual method. Using the Koopman operator we identify a preconditioner for operator equation, which deems essential in our numerical tests. To overcome computational infeasibility we use low rank hierarchical tensor product approximation/tree-based tensor formats, in particular tensor trains (TT tensors) and multi-polynomials, together with high-dimensional quadrature, e.g. Monte-Carlo. By controlling a destabilized version of viscous Burgers and a diffusion equation with unstable reaction term numerical evidence is given.

Mathias Oster, Leon Sallandt & Reinhold Schneider. (2024). Approximating the Stationary Bellman Equation by Hierarchical Tensor Products. Journal of Computational Mathematics. 42 (3). 638-661. doi:10.4208/jcm.2112-m2021-0084
Copy to clipboard
The citation has been copied to your clipboard