Ted Chiang is a writer whose writing is quite exceptional, whether it is technical or fiction. He is, shall we say, a rather careful writer, who does not produce articles and stories rapidly. I pay attention when a new piece of his comes out. He is easily one of my favorite science fiction writers; but his technical pieces written for general audiences are truly outstanding in clarity and accuracy. So, when I saw an article in The Atlantic entitled “No, Artificial Intelligence Is Not Conscious” – an article that even Chrome could predict I might well find interesting – and then realized that Ted Chiang was the author, I wanted to read it immediately. Of course, life interfered with this plan, and it was some hours before I was able to begin reading the article as a passenger while my dear wife Susie drove us home.
I was absorbed and smiling at the shear artistry of Chiang’s descriptions of what continues to pass for “artificial intelligence” these days – that is, large language models (hereinafter designated by the abbreviation “LLM”). His description of LLM’s was to aid in his argument that these models are not conscious, of course, but so well done that they deserve an article and kudos all on their own. That Chiang goes on to carefully explain why today’s LLM’s are not conscious or moral (but still quite useful) reminded me of the kind of illustrations that I first encountered in Douglas Hofstadter‘s writing back when I was one of his students – and then Chiang referred to a comment Hofstadter often made back in those days. This brought a smile. So, at the risk of seeming too much a fan of Chiang – but of course I could not rightly deny such – I decided to recommend this article.
Who should read the article, and why should they read it?
I recommend that anyone who wants to understand the nature of LLM’s in 2026 read this article. The preliminaries of Chiang’s argument include the cleanest and clearest explanation of how LLM’s work and what they are that I have yet read – at least for a general audience.
Secondly, Chiang gets into a more philosophical space as he discusses whether or not LLM’s are conscious or have the capacity to be moral actors. I’ve seen a few nits picked regarding his arguments here; though I found them quite reasonable and plausible. His claims regarding what would be required to truly develop conscious software remind me of speculation I heard from Hofstadter back in the day, as he was completing Gödel Escher Bach – a much more challenging read than this article, of course. I found Chiang’s discussion of machine consciousness to be potentially very helpful to the more casual reader for whom reading GEB would be asking much – but also quite worthy of being mentioned in the context of the larger and deeper work.
Being enamored of the discussion in the article, I decided I would ask Google’s LLM, Gemini, what it thought, er… had to say, er… how it predictively responded to the article. So I did. Below, I reproduce the interaction.
I asked Gemini:
Is Ted Chiang’s article entitled “No, Artificial Intelligence Is Not Conscious,” recently published in The Atlantic, accurate in how it describes the operation of LLM’s?
Gemini’s thorough response, in its entirety, is reproduced below:
Yes, Ted Chiang’s article “No, Artificial Intelligence Is Not Conscious,” published in The Atlantic, is technically and mechanically accurate in how it describes the base operation of Large Language Models (LLMs).
Chiang—noted science fiction author and former computer scientist—avoids overly dense academic jargon and cuts straight to the core mechanics of how these systems function.
His descriptive claims align with how LLMs operate under the hood in several key ways:
1. LLMs as “Sentence-Continuation Machines”
- What Chiang says: He treats LLMs fundamentally as text generators and probabilistic prediction engines rather than feeling, sentient entities.
- The technical reality: This is completely correct. At its core, an autoregressive LLM takes a sequence of input tokens and calculates mathematical probabilities to predict the single most likely next token. It does not “think ahead,” draft a concept in a mind’s eye, and then translate it into prose; it simply extends the statistical pattern of the text it has been given.
2. The Roleplay / Simulation Argument
- What Chiang says: He uses an intuition pump comparing asking an LLM to generate a dialogue between Julius Caesar and Genghis Khan to asking it to roleplay a “helpful assistant”. We intuitively recognize that writing fiction about Caesar does not magically conjure a conscious instance of Caesar; similarly, generating text as a helpful assistant does not mean a conscious entity has woken up inside the machine.
- The technical reality: This accurately reflects instruction-tuning and alignment. An LLM is a chameleon; it adopts a stylistic persona based on prompt conditioning and context windows. When it outputs “I feel…” or “I think…”, it is executing text continuation conditioned on what a helpful human assistant would statistically write in that context, not reporting an internal subjective state.
3. Text as a “Deepfake Medium”
- What Chiang argues: He suggests that we need to view text itself as a deepfake medium—meaning fluent, grammatical, or emotionally resonant language is no longer a reliable proxy for a thinking, feeling mind behind the keyboard.
- The technical reality: This is a vital distinction in the modern AI landscape. Humans have a hardwired evolutionary tendency toward anthropomorphism (attributing minds, intentions, and emotions to things that act communicative). Because LLMs master human syntax and emotional masking so well, they easily trigger our social instincts, making it feel like “someone is home” when mathematically, it is just a high-dimensional matrix multiplication engine outputting text strings.
Where the Essay Focuses (Policy over Metaphysics)
While Chiang’s description of how LLMs operate is accurate, the main thrust of his essay is actually a social and legal critique rather than a pure computer science paper.
He warns that allowing tech companies or the public to blur the lines around AI consciousness creates a convenient moral shield. If people believe a chatbot is a conscious agent that “decided” something of its own volition, it shifts legal and moral accountability away from the corporation that engineered, trained, and deployed it.
In short, Chiang’s technical breakdown of LLMs as sophisticated text-prediction tools—stripped of inner life, intent, or consciousness—is spot on.Yes, Ted Chiang’s article “No, Artificial Intelligence Is Not Conscious,” published in The Atlantic, is technically and mechanically accurate in how it describes the base operation of Large Language Models (LLMs).
Chiang—noted science fiction author and former computer scientist—avoids overly dense academic jargon and cuts straight to the core mechanics of how these systems function.
His descriptive claims align with how LLMs operate under the hood in several key ways:
1. LLMs as “Sentence-Continuation Machines”
- What Chiang says: He treats LLMs fundamentally as text generators and probabilistic prediction engines rather than feeling, sentient entities.
- The technical reality: This is completely correct. At its core, an autoregressive LLM takes a sequence of input tokens and calculates mathematical probabilities to predict the single most likely next token. It does not “think ahead,” draft a concept in a mind’s eye, and then translate it into prose; it simply extends the statistical pattern of the text it has been given.
2. The Roleplay / Simulation Argument
- What Chiang says: He uses an intuition pump comparing asking an LLM to generate a dialogue between Julius Caesar and Genghis Khan to asking it to roleplay a “helpful assistant”. We intuitively recognize that writing fiction about Caesar does not magically conjure a conscious instance of Caesar; similarly, generating text as a helpful assistant does not mean a conscious entity has woken up inside the machine.
- The technical reality: This accurately reflects instruction-tuning and alignment. An LLM is a chameleon; it adopts a stylistic persona based on prompt conditioning and context windows. When it outputs “I feel…” or “I think…”, it is executing text continuation conditioned on what a helpful human assistant would statistically write in that context, not reporting an internal subjective state.
3. Text as a “Deepfake Medium”
- What Chiang argues: He suggests that we need to view text itself as a deepfake medium—meaning fluent, grammatical, or emotionally resonant language is no longer a reliable proxy for a thinking, feeling mind behind the keyboard.
- The technical reality: This is a vital distinction in the modern AI landscape. Humans have a hardwired evolutionary tendency toward anthropomorphism (attributing minds, intentions, and emotions to things that act communicative). Because LLMs master human syntax and emotional masking so well, they easily trigger our social instincts, making it feel like “someone is home” when mathematically, it is just a high-dimensional matrix multiplication engine outputting text strings.
Where the Essay Focuses (Policy over Metaphysics)
While Chiang’s description of how LLMs operate is accurate, the main thrust of his essay is actually a social and legal critique rather than a pure computer science paper.
He warns that allowing tech companies or the public to blur the lines around AI consciousness creates a convenient moral shield. If people believe a chatbot is a conscious agent that “decided” something of its own volition, it shifts legal and moral accountability away from the corporation that engineered, trained, and deployed it.
In short, Chiang’s technical breakdown of LLMs as sophisticated text-prediction tools—stripped of inner life, intent, or consciousness—is spot on.
Long winded, Gemini, to be sure! And, quoting Gemini surely means that a portion of this post used “generative artificial intelligence” – a term to which I object with a mild cringe, nodding to common usage. And so it goes…
A couple of days later, I decided I would ask ChatGPT, the same question. Below, I reproduce ChatGPT’s response, which ChatGPT presented in a way difficult to paste into WordPress.
Yes—but with an important qualification: Chiang is broadly accurate about the basic mechanics of LLMs, but some of the conclusions he draws from those mechanics are stronger than the mechanics themselves warrant.
I found the recent Atlantic essay and checked its central claims against current technical literature, including work published in 2026.Where Chiang is right
The core description is sound:
- An autoregressive LLM is trained primarily to predict the next token from the preceding context.
- A transformer processes the context through layers of learned representations and self-attention.
- At inference, it produces a probability distribution over possible next tokens, selects/samples one, adds it to the context, and repeats.
- There isn’t a little internal sentence-generator or homunculus deciding what to say. The computation is
distributed across enormous numbers of learned parameters.- The fact that the training objective is “predict the next token” is not a metaphor. That’s genuinely the fundamental training objective for standard decoder LLMs.
So if Chiang is arguing against the picture that an LLM is secretly a miniature human mind sitting behind the words, he has a very strong case.
But “just predicting the next word” is misleading
This is where I’d qualify his argument.
“Next-token prediction” describes the training objective, not necessarily the best description of everything the resulting network does internally.
A sufficiently large model can learn internal representations of things that aren’t themselves tokens: syntax, semantic relationships, spatial relationships, facts, abstractions, and apparently even procedures
for solving certain kinds of problems.
Research on transformer mechanics, for example, has found that attention layers can learn mechanisms that amount to retrieving relevant information from context and composing it to produce predictions.
And this is an especially important distinction: Predicting the next token can require constructing an internal representation of the situation that makes the next token predictable.
Suppose I ask: “Alice put the glass on the table. Bob picked up the glass. Where is the glass now?” The model doesn’t necessarily retrieve a memorized sentence that looks like that. It can construct an internal
representation involving Alice, Bob, the glass, the table, and the sequence of events, then use that representation to predict the answer.
That’s still next-token prediction. But calling it merely “autocomplete” can obscure the extraordinary computation that may be occurring in between input and output.
Indeed, a very recent 2026 paper explicitly argues that “just next-token prediction” has become an oversimplification, distinguishing the training objective from the internal computational processes that
emerge in models trained with it.There’s an even bigger problem with Chiang’s argument
I think Chiang’s strongest argument is actually not a technical argument about how LLMs operate. It’s a philosophical argument about consciousness.
And there is an important logical gap: “This system works this way” ® “therefore this system isn’t conscious.” That inference isn’t established by our current science.
We don’t actually know what physical/computational properties are sufficient for consciousness.
Consider a hypothetical: (1) We discover that human brains ultimately perform physical information processing. (2) We discover that some artificial system performs a sufficiently similar kind of information
processing. (3) Does that automatically make the artificial system conscious? We don’t currently know.
Conversely: (1) An LLM is a transformer. (2) It predicts tokens. (3) It doesn’t have a persistent biological nervous system. (4) Therefore it cannot possibly have subjective experience. The last step isn’t something neuroscience has demonstrated.
Philosopher David Chalmers has made essentially this point in discussing LLM consciousness: there are substantial reasons to doubt that today’s LLMs are conscious, but the question isn’t settled merely by
pointing out that they’re language models.And there’s an interesting irony here
Chiang’s essay seems to rely heavily on the distinction between simulation and genuine mental activity. But that’s precisely where things become philosophically difficult.
Imagine a system that has: a persistent internal state; a model of itself; a model of its environment; long-term memory; goals; the ability to plan; the ability to learn from experience; recurrent processing; sensory inputs; and sophisticated self-monitoring.
At what point would saying “it’s merely predicting the next token” cease to be an adequate explanation?
The answer can’t simply be “never,” because we could build a system whose output interface happens to be token prediction while its internal architecture becomes radically more agentic.
And we’re already seeing systems where the LLM is only one component of a larger architecture—memory, tools, retrieval, external state, planning loops, etc. The distinction between the LLM itself and an LLM-based agent therefore matters enormously.One point where I think Chiang is particularly persuasive
His argument that a conversational transcript is extremely weak evidence of consciousness is excellent.
If an AI says, “I’m afraid,” that tells us almost nothing by itself about whether there is an underlying subjective feeling of fear.
The same problem exists in reverse. An entity might conceivably have subjective experience while being unable or unwilling to express it linguistically.
So: humanlike language ¹ evidence of consciousness. That’s a very important corrective to anthropomorphism.My overall assessment
Chiang’s claim My assessment LLMs are fundamentally trained through next-token prediction Correct Transformer attention is crucial to processing context Correct LLMs don’t contain a little conscious “self” directing responses Very likely correct Humanlike conversation doesn’t demonstrate consciousness Correct LLMs are essentially sophisticated autocomplete Misleading if taken literally Next-token prediction explains everything about their cognition Too strong Current LLMs aren’t conscious Plausible and probably correct, but not scientifically proven LLM architecture establishes that machine consciousness is impossible Not established So I’d say Chiang is a very good guide to why we shouldn’t anthropomorphize current LLMs, but a less reliable guide to what can ultimately be inferred about machine consciousness from their
architecture.The really fascinating question, in my view, isn’t “Are today’s LLMs conscious?” I think there’s good reason to answer probably not.
It’s:
If consciousness is an emergent property of information processing rather than something uniquely biological, what additional computational properties would turn an LLM-like system into
something for which “probably not conscious” was no longer a reasonable assumption?
That’s where Chiang’s argument becomes much less definitive—and where the current science gets genuinely interesting.
Like I said, ChatGPT’s response was not presented in a manner that was easy to put into WordPress. So, apologies for the unpleasant formatting above. I trust ChatGPT was not consciously problematic…. [that’s a joke].
So read Chiang’s article, if you find it even a mildly interesting topic.
In closing, I’d like to recommend where else a non-specialist one might turn for practical and reasonable perspectives on artificial intelligence. Arvind Narayanan, Computer Scientist and Professor at Princeton, has produced a number of quite helpful resources. Here are a few recent resources that he produced.
- Debunking Moravec’s Paradox (a 13 minute video on YouTube in 2026)
- AI as Normal Technology (a 50 page paper release in 2025)
- AI Snake Oil (a 2024 book available from the normal places)
That’s all for now.
