Researchers have found that large language models can sometimes detect when a 'thought' is artificially injected into their neural networks, but subsequent studies describe this as anomaly detection, leaving whether it should count as introspection unresolved.
In 2025, Jack Lindsey, a researcher at Anthropic, devised a test of introspection by injecting 'thoughts' into Claude, Anthropic's language model. He dialed up neural activation patterns associated with concepts like 'denim' or 'vegetables' in half of the model copies, while a control group received no injection. Both groups were asked whether they detected an injected thought and what it was about. In trials with the most recent and powerful version, the models correctly flagged the injection 20 percent of the time, with no false positives in the control group.
Language models learn to generate fluent sentences by training on vast amounts of human text, meaning their answers about 'inner life' may simply mimic human language. To avoid this, researchers like Lindsey sought to devise tests that could not be gamed using outside information.
Lindsey's findings sparked a rush of replication attempts. Kyle Mahowald and Harvey Lederman at the University of Texas, Austin, examined whether simpler explanations could account for the results. They separated the two questions, detection and description, and found that the models' vivid descriptions of injected concepts could be explained by the steering technique itself. However, the detection ability proved harder to dismiss. 'The fact that it can notice the anomaly is insane,' Lederman said. 'That is a mind-bending fact.' They called this ability 'anomaly detection.'
Other researchers raised deeper concerns. Shauli Ravfogel of New York University argued that since everything a language model does after reading input involves processing internal states, labeling that as introspection renders the term meaningless. He and his colleagues urged that tests of AI introspection should draw from leading theories of human introspection. One prominent theory defines introspection through second-order mental representations. But applying such distinctions to language models is fuzzy, said Steve Fleming, a neuroscientist at University College London, noting that 'cognitive scientists were in this illusory comfort zone.'
The debate continues. Jacob Andreas of MIT said a maximally deflationary explanation for everything might exist. 'But if there is, I don't think we've actually identified it yet,' he said. Lindsey said, 'Given the state of the theoretical landscape, where no one agrees on anything, I think that our best path forward is to do a lot of empirical science.'
Analysis
Why This Matters
- Challenges how we define intelligence and self-awareness in AI.
- Could influence future AI design and safety considerations.
- Revives long-standing philosophical questions about consciousness in machines.
Background
The study of introspection in AI is new, building on decades of philosophical debate about introspection in humans. Researchers have long debated what introspection means and whether it can be tested in machines. Recent advances in large language models, which can generate human-like text, have made these questions empirically testable. The experiments use techniques to manipulate neural activity, akin to digital neurosurgery.
Key Perspectives
Jack Lindsey (Anthropic): Viewed the 20 percent success rate with no false positives as strong evidence that models can introspect, albeit unreliably.
Kyle Mahowald and Harvey Lederman (University of Texas, Austin): Argue that the models' ability is limited to 'anomaly detection', noticing something unusual in internal processing, but not specific details. They found that the two questions (detect and describe) are separate, and the describe ability is likely due to steering alone.
Shauli Ravfogel (NYU): Raises a broader philosophical concern: if introspection is defined as sensing internal states, then all processing in a language model is internal states, making the term meaningless. He argues tests should be inspired by human introspection theories.
What to Watch
- Further replication studies on other models and at larger scales.
- Development of theoretical frameworks for AI introspection that distinguish first- and second-order representations.
- Potential for researchers to design tests that specifically require second-order awareness.
Checked
11 claims in this story checked against 1 source: 9 supported.
Show each claim
- SupportedLarge language models can sometimes detect artificially injected thoughts in their neural networks.Quanta Magazine: “Lindsey’s experiment demonstrated that models could at least sometimes detect and describe these injections.”
- Partly supportedSubsequent studies describe the detection ability as anomaly detection.Quanta Magazine: “Lederman and Mahowald called this ability “anomaly detection.””
- SupportedWhether detecting injected thoughts should count as introspection remains unresolved.Quanta Magazine: “Others have proved harder to explain away, but whether they should count as introspection remains hotly contested.”
- SupportedIn 2025, Jack Lindsey devised a test by injecting thoughts into language models including Claude.Quanta Magazine: “In 2025, Jack Lindsey, a researcher at Anthropic, devised a way to test introspection in language models by injecting “thoughts” into their neural networks”
- SupportedIn trials with the most recent and powerful version, models correctly flagged injections 20% of the time.Quanta Magazine: “In 20% of trials with the most recent and powerful version, the models correctly flagged that they’d been injected”
- SupportedModels in the control group never reported receiving an injection.Quanta Magazine: “models in the control group never reported receiving an injection.”
- SupportedMahowald and Lederman separated the questions of detecting an injection and describing its concept.Quanta Magazine: “Lederman and Mahowald decided to study the questions separately.”
- Partly supportedThe vivid descriptions of injected concepts could be explained by the steering technique itself.Quanta Magazine: “Since steering alone could make models start talking about the injected concept, Lederman and Mahowald felt that the latter option was more likely.”
- SupportedLederman said, “The fact that it can notice the anomaly is insane. That is a mind-bending fact.”Quanta Magazine: ““The fact that it can notice the anomaly is insane,” Lederman said. “That is a mind-bending fact.””
- SupportedRavfogel argued that calling internal-state processing introspection would make the term meaningless.Quanta Magazine: ““If we say that this is introspection, the term ‘introspection’ becomes kind of meaningless,” Ravfogel said.”
- SupportedFleming said the distinction between first- and second-order representations is fuzzy.Quanta Magazine: ““Going down to a mechanistic level and probing the internals of a large language model makes you realize that that distinction is actually quite fuzzy, and maybe it’s fuzzy in humans as well,” said Steve Fleming”
