Deep Reinforcement Learning - Markov Decision Process (MDP) - Explained (5)

Deep reinforcement learning is positioned as the fix for a core mismatch in finance: supervised learning struggles in ultra high frequency trading because the market environment is highly dynamic, so feature importance and model behavior shift day to day. Reinforcement learning instead learns through active interaction—updating decisions as the environment changes—rather than freezing a model after training. That adaptability is framed as the reason reinforcement learning can outperform “retrain-and-replace” approaches when the same feature set can’t reliably predict future conditions.

The foundation for that adaptive decision-making is the Markov Decision Process (MDP), a mathematical framework for sequential choices under uncertainty. In an MDP, an agent repeatedly observes a state, selects an action from a fixed set, and receives a reward that evaluates the action’s outcome. The environment then transitions to a new state, and the agent continues, aiming to maximize the total (summed) reward over time. Uncertainty is captured by a transition function: taking action a in state s leads to the expected next state with high probability, but there is also a smaller chance of ending up in an unexpected state due to external randomness.

An MDP is built from four components: a set of states, a set of actions, a transition function (often described as probabilities over next states), and a reward function (how much feedback the agent gets for moving between states). For episodic tasks, there is also a start state where interaction begins and a terminal state where the episode ends. Chess is offered as a clean example of an MDP without randomness: the board configuration is the state, legal moves are actions, and the game ends when a terminal condition is reached.

A key simplifying assumption is the “Markov” property: the future depends only on the current state, not on the full history. In other words, once the agent knows the present state, earlier states don’t add extra predictive power for rewards or transitions. This makes the problem tractable and turns decision-making into a state-based optimization.

The transcript grounds these ideas with a car overheating example. The environment has three states—cool, warm, and overheated—with actions “move slow” and “move fast.” Moving slow from cool yields +1 reward and keeps the car cool with 100% probability, while moving fast from cool gives +2 reward but carries a 50% chance of transitioning to warm. In warm, moving fast guarantees overheating and a -10 penalty, while moving slow has a 50% chance to cool back down. The optimal policy follows a simple rule: in cool, move fast; in warm, move slow; once back in cool, move fast again. When the car reaches the overheated terminal state, the episode ends, and a new episode begins—allowing reinforcement learning to improve behavior over repeated trials.

Overall, the MDP formalism provides the structure that lets reinforcement learning learn adaptive policies through trial-and-error interaction—an approach presented as especially relevant when static supervised models can’t keep up with shifting environments like financial markets or other real-world systems.