强化学习研究为何普遍假设马尔可夫性？
Great question—this is something a lot of folks grapple with once they move beyond the basics of RL. Even though we know the strict Markov property rarely holds in real-world scenarios, here are the core reasons the MDP framework remains the backbone of most RL research:
Mathematical Tractability & Foundational Theory: The MDP framework gives us a rock-solid, decades-old theoretical foundation. Bellman equations, value/policy iteration, convergence guarantees—all these core tools rely entirely on the Markov property. Without it, we lose most of the formal math that lets us analyze why an algorithm works (or doesn’t) and prove things about its behavior. Non-Markovian settings (like POMDPs or history-dependent environments) get messy fast: you can’t just write a clean Bellman update, and defining clear, generalizable objectives becomes way harder. Researchers start with MDPs because they provide a stable, understandable base to experiment with core ideas (like policy gradients or Q-learning) without getting bogged down in the complexities of modeling every bit of history.
Pragmatic State Space Approximations: In practice, we often work around non-Markovianity by expanding the state representation to include relevant historical context. For example, if a robot’s current sensor data doesn’t capture enough to predict the next state, we can append the last 2-3 sensor readings to the state vector. This turns a non-Markov problem into an approximate MDP, and more often than not, this works surprisingly well. It’s a practical workaround that lets us reuse existing MDP-based algorithms instead of reinventing the wheel for every unique non-Markov scenario.
Standardized Benchmarking: The RL community has invested heavily in building standardized MDP-based benchmarks—think Atari games, MuJoCo robotics tasks, or GridWorld environments. These benchmarks let researchers compare algorithms on a level playing field. If we shifted to non-Markovian setups, there’s no universal way to model history dependency (how much history? Which parts are relevant?), which would make comparing results nearly impossible. Sticking to MDPs ensures consistency and accelerates progress by letting everyone build on the same set of test cases.
Computational Efficiency: Tracking historical states blows up the state space exponentially. For example, if each state has 10 possible values, adding just 3 steps of history makes the effective state size 10^4 instead of 10. This increases computational cost dramatically, especially for deep RL models that require millions of training steps. MDPs keep the state space manageable, which is critical for scaling algorithms and testing them in reasonable timeframes.
Incremental Progress Culture: Most RL research focuses on incremental improvements to existing methods. Since MDPs are the default framework, building on top of them allows researchers to iterate quickly—tweaking a policy gradient algorithm or improving Q-function approximation, for example. Tackling non-Markovian problems is a more specialized, high-effort area, so it’s often treated as an extension rather than the starting point.
It’s worth noting that work on non-Markovian RL (like using recurrent networks to model history or POMDP solvers) does exist, but it’s often niche compared to MDP-based research because of the above constraints. The MDP assumption isn’t just a lazy choice—it’s a pragmatic one that balances theory, practicality, and progress.
内容的提问来源于stack exchange，提问作者niko
