Reinforcement Learning: An Introduction

A comprehensive foundational textbook on reinforcement learning, covering key algorithms such as Q-learning, Sarsa, and TD-learning, while bridging the gap between tabular methods and function approximation.

Master the definitive science of goal-directed learning from interaction.

Author: Richard S. Sutton and Andrew G. Barto

Acknowledgments: Supported by the Air Force Office of Scientific Research, the National Science Foundation, and GTE Laboratories.

1. Define Reinforcement Learning and distinguish between immediate rewards and long-term value functions.
2. Identify and describe the four sub-elements of a reinforcement learning system.
3. Apply the Temporal-Difference (TD) update rule to a state-value lookup table.
4. Implement incremental update rules for both stationary and nonstationary reward distributions.
5. Evaluate the effectiveness of Optimistic Initial Values and Upper-Confidence-Bound (UCB) action selection in promoting exploration.
6. Explain the mechanics of Gradient Bandit algorithms using numerical preferences and softmax distributions.
7. Define the agent–environment interface and model complex tasks using the MDP framework.
8. Calculate expected returns for both episodic and continuing tasks using discounting and unified notation.
9. Distinguish between state-value (v_\pi) and action-value (q_\pi) functions and derive them using Bellman equations.
10. Perform Policy Evaluation to compute state-value functions for arbitrary policies using iterative methods.