1. At a Glance

Official page (Sutton & Barto, verified 2026-07-25):
http://incompleteideas.net/book/the-book-2nd.html

2. Why This Book

Ask a Specialist-tier engineer why a humanoid robot’s whole-body controller picked one recovery motion over another, and "it’s the AI" will not satisfy an OEM support call. Reinforcement Learning: An Introduction is the book that turns that question into something a service engineer can actually investigate. Thrun’s Probabilistic Robotics (Top 10, No. 7) covers the estimation half of uncertainty — where the robot is, what the sensor data means. Sutton and Barto cover the decision half: given an estimate of the world, how an agent picks actions, and how that choice was trained.

Robotics Service Framework Professional treats an AI-driven fault as a black box — reset, reboot, escalate — which is the right call for an eight-module, 36-hour entry program. Specialist-tier engineers working Module 7.2 humanoid faults need more, because OEM documentation for VLA (vision-language-action) models and whole-body controllers is written in this book’s vocabulary: reward, policy, value function, exploration. Without that vocabulary, a call with an OEM’s AI support team about unexpected behavior stays opaque. The book is free, it is the field’s most cited reinforcement learning text, and it is the shared language between service engineers and the teams who build the policies they end up servicing.

3. What’s Inside

The book is organized to build the theory from the simplest possible learning problem upward. It opens with multi-armed bandits — action selection under uncertainty with no state — before introducing the finite Markov decision process (MDP), the framework the rest of the book builds on: states, actions, rewards, and the goal of maximizing cumulative reward. From there it develops three families of solution methods and shows how they relate: dynamic programming (solving an MDP when the model is fully known), Monte Carlo methods (learning from complete episodes with no model), and temporal-difference learning (learning from incomplete episodes, bootstrapping estimates from other estimates) — including the two algorithms every practitioner ends up naming, Q-learning and SARSA.

The middle chapters unify these into n-step bootstrapping and planning-and-learning methods that combine a learned model with direct learning from experience (Dyna). The second half of the book turns to function approximation, needed once the state space is too large to represent as a table — on-policy and off-policy prediction and control with function approximation, eligibility traces, and policy gradient methods, which learn a parameterized policy directly rather than deriving it from a value function. The book closes with case studies (TD-Gammon, the Watson Daily-Double wagering strategy, and others) that show the theory applied end to end. This is the 2nd edition (2018), substantially expanded from the 1996 original to cover the deep-RL era, and it remains free.

4. The RSF Perspective

Robotics Service Framework Module 7.2 names "VLA/embodied AI models" and "whole-body controller" as the technology stack behind humanoid robots, and this book is the foundation under any learned control policy a service engineer will meet in that stack. A VLA model is, in this book’s terms, a policy — a function from observation to action, trained to maximize some reward signal. A whole-body controller that blends a learned component with classical control is doing, in part, what the policy-gradient chapters describe: optimizing a parameterized action distribution against experience. Reading this book does not let a service engineer retrain that policy, and it does not need to. What it does is convert "why did the robot do that" from a shrug into a specific, answerable question.

Three questions recur in Module 6’s AI-layer diagnosis work, and this book supplies the vocabulary for all three. Was the reward shaped wrong — is the robot optimizing for something adjacent to the intended behavior, a known failure mode this book introduces as reward hacking? Is the policy out of distribution — is the robot in a state its training data never covered, which Part I’s discussion of exploration versus exploitation explains from the ground up? Or is this an exploration artifact — a still-exploring policy taking a probing action that looks like a fault but is not? An engineer who can ask an OEM’s AI support team "is this a reward-shaping issue or a distribution-shift issue" gets a materially different conversation than one who can only say "it did something weird."

Reading protocol: Part I — the chapters running from multi-armed bandits through tabular temporal-difference learning and Dyna-style planning — is the conceptual foundation and is worth reading in full; it is self-contained and does not require the deep-learning background the second half assumes. The function-approximation and policy-gradient chapters (Part II) are Expert-tier territory: they explain how modern VLA policies are actually trained, but they assume comfort with gradient-based optimization and are not needed to have the Module 6/7.2 diagnostic conversation described above. Skip them on a first pass and return once committed to AI/perception specialization rather than general humanoid service work.

5. Difficulty & Audience

Difficulty splits sharply by half. Part I (roughly the first eight chapters) sits at four of five stars: the mathematics is precise but self-contained, building from single-state bandits to full tabular MDP solution methods with worked examples throughout. Part II (function approximation onward) is five of five — it assumes working comfort with linear algebra and gradient-based optimization, and the notation does not slow down for a first-time reader. Budget roughly 30–40 hours for Part I alone; the full book, worked seriously, is a multi-month undertaking, not a weekend read.

This book is not required at RSF Professional tier — the 36-hour Professional program does not test policy-gradient math or MDP formalism. Read it when specializing under Module 7.2 (humanoid) or, more broadly, Module 6 AI-layer diagnosis, where "why did the AI do that" becomes a routine service question rather than an edge case.

6. Companions & Alternatives

Pair it with Thrun, Burgard and Fox’s Probabilistic Robotics (Top 10, No. 7): Thrun covers estimation under uncertainty, this book covers decision-making under uncertainty, and Module 6’s AI-layer diagnosis work draws on both. For a shorter, more current complement, Mykel Kochenderfer’s Algorithms for Decision Making (free online) covers overlapping ground with more recent methods and a stronger emphasis on decision-making under partial observability, useful once Sutton and Barto’s tabular foundation is solid. Neither is a substitute for the other — read Sutton and Barto first for the concepts, then Kochenderfer for how the field has moved since.