Winda

Carnegie Mellon 11-768: AI Agents

Instructions

This course studies LLM-based AI agents: systems that use large language models to perceive, reason, plan, and act over many steps. Agents interact with their environments — calling tools, navigating interfaces, writing code, working alongside people — to complete real tasks. Course website: https://www.cmu-agents.com/ Instructors: - Daniel Fried - Graham Neubig

Contents · 31

  1. 01-Course Overview - What Is an AgentOpening lecture for a course on AI agents, introducing instructors and topics, with examples of agent successes and failures and a class discussion on when agents can be trusted to act autonomously.
  2. 01-Course Overview - What Is an AgentAn introductory lecture defining AI agents, covering capabilities, tools and tool calling, the agent loop (ReAct), and building capabilities through LLM training versus harness engineering.
  3. 02-Agent Capabilities 1 - Tool UseHow language model agents invoke external tools: defining tool interfaces, representing tool calls as tokens across model protocols, dispatching and validating calls, and composing actions programmatically with code as a meta-tool.
  4. 02-Agent Capabilities 1 - Tool UseLecture on tool use for language model agents, covering what tools are, why they extend or facilitate LLM capabilities, and common tool categories like search, code execution, and APIs.
  5. 03-Agent Capabilities 2 - Context Management for Long-Context AgentsContext management for long-context agents: why agent contexts grow quadratically, what composes them in practice, and the inference challenges of capacity, efficiency, KV caching, and scheduling.
  6. 03-Agent Capabilities 2 - Context Management for Long-Context AgentsHow agent context grows and strains LLM inference: serving-stack phases and metrics, attention refresher, then long-context architectures like sliding-window, sparse, and linear attention, DeltaNet variants, and KV compression.
  7. 04-Agent Capabilities 3 - Skills and MemoryLecture on skill induction and agent memory: how agents store knowledge across tasks in external stores, reuse learned subtask skills, and manage memory lifecycles, compared to in-context retention and weight updates.
  8. 04-Agent Capabilities 3 - Skills and MemoryA lecture on how agents accumulate reusable knowledge across tasks, covering authored skills like SKILL.md files, progressive disclosure, skill induction from episodes, and measured gains from SkillsBench.
  9. 05-Agent Capabilities 4 - Planning, Task Decomposition, and Multi-Agent CoordinationAn introduction to planning and task decomposition for agents, covering plan representations, revising plans based on environment feedback, replanning, parallel task structures, and trade-offs between commitment and flexibility.
  10. 05-Agent Capabilities 4 - Planning, Task Decomposition, and Multi-Agent CoordinationHow LLM agents plan and decompose tasks: when planning helps or hurts, plan representations like chain-of-thought and least-to-most prompting, decomposition, reasoning versus acting, and classical planning.
  11. 06-Domains 1 - Coding AgentsLecture transcript on building coding agents, covering how language models are trained to write, edit, and reason about code, including pre-training data, tokenization, and fine-tuning considerations.
  12. 06-Domains 1 - Coding AgentsCovers LLM-based coding agents and how models that write code are trained, from pre-training on code corpora and mid-training mixtures to infilling, plus evaluation methods for generated code including reference matching and execution-based testing.
  13. 07-Domains 2 - Computer Use AgentsAn introduction to computer use agents, covering what they are, how they differ from other agents, and how they operate by observing screenshots and executing human-like actions such as clicking and scrolling in graphical interfaces.
  14. 07-Domains 2 - Computer Use AgentsOverview of computer use agents that operate GUIs through an observe–reason–act loop, tracing their evolution from toy web environments to realistic desktop tasks, and surveying benchmarks from static action prediction to end-to-end and long-horizon evaluation.
  15. 08-Training 1 - Supervised Fine-Tuning (SFT)An introduction to training agents by updating model weights, covering supervised fine-tuning as a cold-start stage before reinforcement learning, what trajectories teach models, and why SFT remains necessary.
  16. 08-Training 1 - Supervised Fine-Tuning (SFT)How recorded agent trajectories become model weights: loss masking over assistant tokens, chat templates and data formats, the SFT-to-RL pipeline, and how to evaluate whether the fine-tuned agent actually improved.
  17. 09-Training 2 - Reinforcement Learning BasicsIntroduction to reinforcement learning for agents, bridging from SFT to policy gradient methods, illustrated through a number-guessing task and covering rewards for trajectories.
  18. 09-Training 2 - Reinforcement Learning BasicsIntroduces reinforcement learning fundamentals for agents, contrasting SFT with RL to address task, data, and exposure mismatches; covers agent-environment interactions, trajectories, expected reward objectives, reinforced self-training, and policy gradients.
  19. 10-Domains 3 - Deep Research AgentsLecture by an assistant professor introducing deep search agents: why complex information needs require agentic systems beyond simple RAG, how benchmarks like BrowseComp evaluate them, and how to build agents and retrieval systems.
  20. 10-Domains 3 - Deep Research AgentsLecture introducing deep research agents that perform multi-step research for long-form, citation-backed answers, covering the tasks they address, benchmarks like ScholarQA, and training methods such as reinforcement learning with evolving rubrics.
  21. 11-Training 3 - Advanced RL AlgorithmsCovers advanced RL algorithmic techniques for training agents, using a coding-agent bug-fixing running example. Discusses credit assignment, dense rewards, reward-to-go, and improving on REINFORCE/GRPO's high variance and coarse feedback signals.
  22. 11-Training 3 - Advanced RL AlgorithmsCovers advanced reinforcement learning for agents: credit assignment via baselines and reward-to-go, turn-level rewards, and value functions learned with critics, using a bug-fixing coding agent as the running example.
  23. 12-Training 4 - RL SystemsAn overview of reinforcement learning systems for training LLM agents, covering core operations like rollouts and optimizer steps, the split between inference engines and trainer engines, and why specialized frameworks exist.
  24. 12-Training 4 - RL SystemsCovers LLM reinforcement learning training systems: the train-inference split between trainer and rollout engines, the training step loop, and optimizations like KV/prefix caching, continuous and speculative decoding, activation checkpointing, and weight sync.
  25. 13-Safety 1 - Agent SafetyReal-world agent safety failures and attacks, including destructive actions, data exfiltration via prompt injection, supply chain compromises, and malicious actors using agents to steal payment data.
  26. 13-Safety 1 - Agent SafetyReal incidents of agent harm, threat models like prompt injection and malicious users, defenses through red teaming and safety training, and sandboxing techniques to limit what agents can access and do.
  27. 14-Frameworks 1 - OpenHandsA lecture on building OpenHands, an open-source agentic coding assistant, contrasting it with tools like Claude Code and Codex and covering its history, features, automations, and lessons for turning user pain points into research projects.
  28. 14-Frameworks 1 - OpenHandsOpenHands coding agent framework: evolution from code completion to SWE-bench and Devin, the OpenHands SDK (tools, conversations, persistence, condensers, Docker execution), and single- versus multi-agent design with skills and MCP tools.
  29. 15-Frameworks 2 - Orchestration and MonitoringCompares single agents with explicit workflows, covering workflow benefits like consistency and efficiency, LangGraph implementation of a support pipeline with routing, shared state, node functions, graph assembly, and recoverable execution.
  30. 16-Agents and the Future of WorkOverview of AI agents for human work: what agents can and cannot do, how benchmarks, environments, and evaluation are built for work tasks, and the technical and societal challenges of aligning agents with real-world jobs and human collaboration.
  31. 17-Interaction 1 - Multi-Agent InteractionIntroduces multi-agent systems for LLM agents, covering role-based decomposition, multi-agent debate, and open-ended delegation with subagents, plus implementation with orchestration frameworks and coordination primitives.