Kosoy argues AI alignment needs deep theory, not trial and error
She defines the technical problem as building superhuman AI that stays aligned with human intentions.
AGI & Safety#05
AGI & Safety
Image: AI Alignment Forum
The short version
- Vanessa Kosoy defines the technical AI alignment problem as designing an AI system that brings superhuman capability to multi-domain, long-term planning while staying aligned with user intentions and preferences.
- She argues that the minimal task for safe AI is creating a reliable global defense system against dangerous AI.
- She argues that the incremental approach used in the AI industry fails if the problems lead to AI takeover.
- She says aligned AI needs aligned preferences, beliefs and decisions, and that philosophical questions about these are not easily brushed aside.
Why it matters
The essay frames alignment as a requirement for AI that can reason about the whole planet and plan for decades or centuries. It argues that finding problems after deployment is too late if the problems lead to takeover. For developers, it means trust depends on beliefs and decisions, not only on goals.
“The problem of making AI safe is underspecified if we haven't decided what the AI is supposed to do”