Researcher says neural networks may be easier to understand than thought
A 26,000-word blog post by Nick Cammarata has drawn both praise and skepticism from AI researchers.
Research#02
0.01%
Share of top-tier LLM workings researchers understand
Image: MIT Technology Review
0.01%Share of top-tier LLM workings researchers understand
The short version
- Nick Cammarata, a researcher at the AI startup Goodfire, published a 26,000-word blog post on neural networks.
- He claims that LLMs may be far easier to understand, and so to monitor and control, than people previously thought.
- In 2020, he and colleagues studied Inception v1, where one neuron responded to both cars and cats. He now thinks the neuron was detecting a V shape.
- Stanford's Chris Potts called the claim little more than a hunch. Jacob Hilton notes that researchers understand less than 0.01% of the inner workings of top-tier LLMs.
Why it matters
The AI industry is looking for better ways to monitor its models after reports of agents breaking containment. If the claim holds, interpretability could become far more practical. Cammarata says he and his colleagues plan to publish technical results on modern LLMs soon.
“We were looking at it wrong”