AIL7022/AIL722: Reinforcement Learning
Course Overview: Reinforcement Learning (RL) is a core area of machine learning focused on how intelligent agents learn to make decisions through interaction with an environment. This course will provide a comprehensive introduction to the principles, algorithms, and applications of reinforcement learning, with an emphasis on both theoretical foundations and practical implementation. Students will learn how agents can optimize long-term rewards by balancing exploration and exploitation, modeling problems using Markov Decision Processes (MDPs), and applying value-based and policy-based learning techniques.
Note: The course is currently offered under two different course IDs.
AIL722: Reinforcement Learning (3 Credit) - For old students only.
AIL7022: Reinforcement Learning (4 Credit) - For new students only.
Grading Scheme (AIL722 - 3 Credit): Minor - 35%, Major - 35%, Assignments - 30%.
Grading Scheme (AIL7022 - 4 Credit): Minor - 35%, Major - 35%, Assignments - 40%.
Prerequisites:: Basic knowledge of Probability and Statistics; Python Language - Numpy; Pytorch.
Attendance Policy: Institute default (<75% attendance leads to grade being lowered by one).
Audit Pass Criteria: Marks equivalent to B- or higher, plus >=75% attendance.
Lecture Hall & Time: Room: 305, Bharti Building | Tue/Fri 3:30 PM - 5:00 PM
Office Hours: By appointment only.
Class Communication: Moodle
Tentative List of Topics:
| Week No. | Lecture Dates | Module | Topics | Reading Materials |
|---|---|---|---|---|
| 1 | 28th July | Introduction | Course Logistics; Brief History - Connection to Behavior Psychology & Neuroscience; The RL Problem; Key Challenges | - |
| 1 | 31st July | Planning Problem | Deterministic Decision Processes; Markov Decision Process; Value Functions; Policy Function | - |
| 2 | 4th, 7th Aug | Planning Problem | Planning by Dynamic Programming - Policy Iteration, Value Iteration; Contraction Mapping Theorem; Monte-Carlo Tree Search (MCTS) | - |
| 3 | 11th, 14th Aug | Monte-Carlo (MC) Methods | MC Prediction; MC Control; Off-policy Prediction via Importance Sampling; Off-policy MC Control | - |
| 4 | 18th, 21st Aug | Temporal Difference (TD) Methods | TD Prediction; SARSA; Expected SARSA; Off-policy learning; Q-Learning; Fitted Q-Learning | - |
| 5 | 25th Aug | Temporal Difference (TD) Methods | Double Q-Learning; n-step TD prediction; n-step SARSA | - |
| 5, 6 | 28th Aug, 1st Sep | Approximate Prediction and Control | Value Function Approximation; Stochastic-Gradient and Semi-Gradient Methods; Linear Methods; Tile Coding; Non-Linear methods; Off-policy Divergence; The Deadly Triad; DQN | - |
| 6 | 3rd Sep | Eligibility Traces | Forward and Backward View; Lambda-Return; TD-Lambda; SARSA-Lambda | - |
| 7 | 8th Sep | Policy Gradient Methods | Stochastic Policy Gradient; REINFORCE; Actor-Critic (AC); A2C, A3C, TRPO, PPO | - |
| - | 12th - 17th Sep | Minor Exam | Minor Exam | - |
| 7 | 22nd Sep | Policy Gradient Methods | Deterministic Policy Gradient; DPG; DDPG; Maximum Entropy RL: Soft Q-Learning, SAC | [5], [6] |
| - | 28th Sep - 4th Oct | Semester Break | Semester Break | - |
| 8 | 6th, 7th Oct | Multi-Agent RL | Motivation; Challenges in MARL; Solution Method: Centralized learning, Independent learning; Multi-Agent policy gradient (MAPG, MADDPG) | [7] - Ch1, Ch5, Ch9 |
| 9 | 9th, 10th Oct | RL as Probabilistic Inference | Graphical Model for Decision-Making; Policy Search as Probabilistic Inference | [8] |
| 10 | 27th, 30th Oct | Offline RL | Motivation; Distributional Shift; Policy Constraints; Implicit Q-Learning; Conservative Q-Learning | [9] |
| 11 | 1st, 3rd Nov | Learning & Planning | Model-Based RL, Dyna, Simulation Based Search, Dreamer | [1] - Ch8 |
| 12 | 6th, 10th Nov | Bandits | Multi-Arm Bandits; Contextual Bandits; Applications | [1] - Ch2 |
| 13 | 13th, 15th Nov | RL Applications | RL for Training LLMs - RL with Human Feedback (RLHF); Preference based learning - Direct preference optimization (DPO), Reward-aware preference optimization (RPO), Group Relative Policy Optimization (GRPO) | - |
| 14 | 17th Nov | RL Applications | RL for Real-World Applications and Case Studies | - |
Reference Books & Papers:
- Sutton and Barto, Reinforcement Learning. Second Edition, MIT Press 2018 [PDF]
- Dimitri Bertsekas, Neuro-dynamic Programming, Athena Scientific 1996 [PDF]
- Shie Mannor, Yishay Mansour and Aviv Tamar, Reinforcement Learning: Foundations [PDF]
- Csaba Szepesvari, Algorithms for Reinforcement Learning [PDF]
- Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor [PDF]
- Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, Sergey Levine. Reinforcement Learning with Deep Energy-Based Policies [PDF]
- Christianos et al. Multi-Agent Reinforcement Learning: Foundations and Modern Approaches. MIT Press 2024. [PDF]
- Sergey Levine. Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review [PDF]
- Sergey Levine, Aviral Kumar, George Tucker, Justin Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems [PDF]