Information and Stochastic Optimization Problems

In Chap. 2, we presented static stochastic optimization problems with open-loop control solutions. In Chap. 3, we introduced various tools to handle information. Now, we examine dynamic stochastic decision issues characterized by the sequence: information → decision → information → decision → etc. This chapter focuses on the interplay between information and decision. First, we provide a “guided tour” of stochastic dynamic optimization issues by examining a simple one-dimensional, two-period linear dynamical system with a quadratic criterion. We examine the celebrated Witsenhausen counterexample, then describe how different information patterns deeply modify the optimal solutions. Second, we present the classical state control dynamical model. Within this formalism, when an optimal solution is searched for among functions of the state, optimization problems with time-additive criterion can be solved by Dynamic Programming (DP), by means of the well-known Bellman equation. This equation connects the value functions between two successive times by means of a static optimization problem over the control set and parameterized by the state. This provides an optimal feedback. We conclude this chapter with more advanced material. We present a more general form of optimal stochastic control problems relative to the state model. Following Witsenhausen, we recall that a Dynamic Programming equation also holds in such a context, due to sequentiality. This equation also connects the value functions between two successive times by means of a static optimization problem. However, the optimization is over a set of feedbacks, and it is parameterized by an information state, the dimension of which is much larger than that of the original state.