Kzzzza 中

RL Personal Learning Path

554 words 1 min #RL

This page is machine-translated and may contain errors. Read the Chinese original

A reference learning path for reinforcement learning (oriented toward using RL to solve robot control problems)

Prerequisites recommended before starting:

  • Linear algebra fundamentals
  • Calculus fundamentals
  • Probability theory and stochastic processes fundamentals
  • Python fundamentals
  • Basic understanding of machine learning algorithms
  • Basic understanding of deep learning
  • Basic familiarity with Linux (if you need hands-on code engineering practice)

This article includes:

  • Recommended learning materials and courses
  • My personal learning path (specifically what content I studied)

Note: this article does not provide electronic copies of textbooks. If you need them, please purchase legitimate copies yourself

1. Recommended books and reference textbooks for learning the fundamental theory:

1. Mathematical Principles of Traditional Reinforcement Learning: Starting from MDPs to Understand the Underlying Mathematical Principles of Algorithms:

Recommended textbook and course: Westlake University's "Mathematical Principles of Reinforcement Learning" — Professor Zhao Shiyu

2. Deep Reinforcement Learning: DeepMind's AlphaGo, the PPO proximal optimization algorithm most commonly used in robotics, and high-dimensional state spaces

  • Recommended textbook: the "Cat Book" Deep Reinforcement Learning (Wang Shusen)

  • Companion code repository: https://github.com/DeepRLChinese/DeepRL-Chinese.git (appears incomplete, some code is missing)

  • bilbil reference course videos: Deep Reinforcement Learning Online Course

  • An RL classic, but since it covers too much and most of it won't be used, treat it as a theoretical reference: Reinforcement Learning: An Introduction (Sutton)

  • A good RL survey — even just skimming the table of contents helps a lot with grasping the overall landscape of reinforcement learning
    "Reinforcement Learning: An Overview" —— arXiv 2412.05265

2. Recommended Code Practice Resources:

  • For basic code practice, Hands-On Reinforcement Learning is recommended.

3. Suggested Learning Path for Getting Started Quickly:

Robot control is a special application problem of reinforcement learning (continuous action space and state space)
Currently, reinforcement learning methods for WBC (whole-body control of humanoid robots) mainly use the PPO algorithm
To get started quickly, it is recommended to combine the materials above and follow the path below:

  1. Basic theory:
    1. Step 1: Mathematical Foundations of Reinforcement Learning:
      1. Just going through the textbook once is enough
      2. It covers most of the theoretical foundations
      3. Chapters 1 through 10 are progressive, the textbook is written very clearly, and it also indicates which parts are extended mathematical derivations that can be skipped
    2. Step 2: PPO extension
      1. Mathematical Foundations of Reinforcement Learning only covers up to the A2C algorithm; PPO and TRPO are improvements over A2C, so some additional knowledge is needed
      2. The basic idea of this part (trust region policy optimization) can be found in Deep Reinforcement Learning, Section 9.1
      3. For TRPO, you only need to know the basic idea; there is no need to worry about code implementation or specific mathematical derivations (they are too complex and difficult to solve)
      4. For PPO, you need to know how the mathematical formulas are derived and also be able to write the code; refer to Hands-on Reinforcement Learning, Chapter 12
  2. Code implementation:

Most reinforcement learning implementations in current project code use packaged libraries (for example, Beyondmimic uses RSL-RL), so the code practice here is only to help understand the algorithm principles:

Referring to Chapters 9, 10, and 12 of Hands-on Reinforcement Learning is enough

If you can fully understand this code, then you will at least have grasped:

    1. Monte Carlo (MC) methods
    2. Temporal difference (TD) algorithms
    3. Value functions
    4. REINFORCE policy gradient
    5. QAC
    6. A2C (Actor-Critic with baseline)
    7. SA (stochastic approximation theory)
    8. Trust region policy gradient
    9. PPO algorithm