Engineering Notes
Thoughts and Ideas on AI by Muthukrishnan
Tag: Alignment
Corrigibility and Interruptibility Building Agents That Accept Human Override
How to design AI agents that accept correction, allow safe interruption, and remain under human control even as their capabilities growRLHF and Preference Learning Teaching Agents What Humans Actually Want
Master reinforcement learning from human feedback — the algorithm behind ChatGPT and modern aligned agents — from reward modeling and PPO to Direct Preference Optimization.Inverse Reinforcement Learning Inferring Goals From Behavior
Learn how inverse reinforcement learning lets AI agents discover hidden reward functions by observing expert behavior, and why it matters for agent alignment and autonomous systemsEnsuring Agent Safety with Constitutional AI Guardrails
## Concept Introduction **Constitutional AI (CAI)** is a method for training an AI to supervise itself. Rather than re...