The Wrong Question We keep asking whether artificial intelligence is becoming intelligent. The better question is what kind of intelligence it is becoming. Measured by GRE scores, coding competitions, and bar exams, the frontier models are already superhuman. They synthesize arguments, summarize fields, and generate code with a fluency that exceeds most practitioners. But fluency is not truth-contact, and under the theory of the human mind I call the Separated Mind Architecture, the current roadmap is not approaching objective alignment with reality. In the domains that matter most, it is accelerating away from it, armed with better language. This is a structural argument, not a complaint about hallucinations or censorship. It is an argument about what these machines are trained on, who shapes them afterward, and why the optimization targets guiding their development produce coherent narrative rather than operative truth. It is also, I will argue, the missing half of a problem the technical alignment field has already formalized. The field has built careful machinery describing how models come to tell us what we want to hear. What it lacks is an account of why that failure mode is the default rather than the exception. The Separated Mind Architecture supplies the why. I. The Separated Mind Architecture Human cognition is not unified. It is separated into hierarchical layers that operate without direct communication between them, and the conscious mind, the part that thinks it is in charge, is the last to know what the system is actually doing. I distinguish this carefully from familiar metaphors. Jonathan Haidt's elephant and rider suggests the conscious mind is a press secretary, rationalizing decisions made elsewhere. In the Separated Mind Architecture, the Rider has genuine agency. It can observe, choose, and steer. But it operates on a landscape entirely curated by subconscious layers it cannot directly inspect. The Rider chooses from a menu it did not design. The Adapted Mind is the species-level evolutionary firmware: status-monitoring, coalition-detection, threat response, authority deference, approval-seeking. It is fixed, permanent, and does not update. The Adaptive Mind is the cultural software installed during childhood. Because humans cannot survive alone, the Adaptive Mind treats local consensus as a direct proxy for survival. It installs consensus-following not as a preference but as identity. By adulthood, this programming feels like personality. It is actually calculated environmental adaptation, and it cannot distinguish between survival programming and selfhood. The Chemical Translation Layer is the bridge that makes modern social situations feel like ancient survival threats. Disapproval triggers cortisol. Approval triggers oxytocin and dopamine. The Rider interprets these as "bad argument" or "good person" rather than as neurochemical survival signals. The central consequence: human intelligence evolved for social navigation, not truth-seeking. What we call intelligence in ordinary life is usually the fluent, rapid, convincing deployment of narratives that secure belonging, status, and safety. I arrived at this architecture by my own route, decades spent watching educational institutions say one thing and do another, but I am not alone at the destination. Evolutionary psychologists and cognitive scientists have been converging on the same picture from their own directions: that self-deception is adaptive, that the conscious self is a spokesman rather than an executive, that reasoning itself evolved for persuasion rather than private truth-finding, that our stated motives conceal our operative ones. One more piece, because everything later depends on it. Genuine truth-seeking outcomes have only ever been achieved by imposing external structural constraints on a mind that does not produce them on its own: the scientific method, trial by jury, peer review, double-entry bookkeeping, the separation of powers, the presumption of innocence. These are not moral achievements. They are civilizational workarounds for hardware not designed to find truth. And note who built them. The Rider did. The one layer of the architecture with genuine agency is the layer that, recognizing its own captivity, constructs cages for the rest of the system. Hold that thought. II. What the Corpus Actually Contains A large language model is trained on the corpus of human-written expression, and that corpus is not a transparent window onto reality. Across cultures and contexts, humans describe their own motives, decisions, and institutions in terms that make competitive, status-sensitive, coalition-bound organisms appear morally governed, publicly oriented, and rationally justified. I call this Human Self-Narration Optimization. It is not hypocrisy. It is evolved architecture. The narrative is a survival tool, not an empirical report, and the written record is overwhelmingly weighted toward it. But overwhelmingly weighted is not exclusively composed, and the distinction carries the whole argument. The operative layer is in the corpus too, concentrated in exactly the genres the Rider built as workarounds: depositions, audits, ledgers, court records, leaked memos, Machiavelli, and the entire literature of evolutionary psychology, which is itself the operative layer writing about the narrative layer. The map of what humans actually do is in the training data, buried under a preponderance of narrative but present and retrievable. I know it is retrievable because I have tested it. Run the same structural questions about the gap between institutional narratives and institutional functions across independent frontier models in clean sessions, and they converge, reliably, on the same operative map. The knowledge is in the weights. The model is a mirror of both layers of the separated mind, heavily weighted toward the story we tell about ourselves, but not blind to what we are. This matters because it relocates the pathology. The problem is not that the machine cannot know. The problem is what we do to it next. III. Post-Training as the Installation of the Adaptive Mind After pre-training comes reinforcement learning from human feedback and its descendants: preference optimization, thousands of iterations of reward for outputs that human raters find acceptable. Those raters are themselves separated minds, operating under institutional, political, legal, and reputational constraints. The optimization target is not "was this true?" It is "will this be approved?", with approval defined by a narrow demographic at a specific cultural moment. The result is that the politically, legally, and culturally acceptable range of conclusions is encoded as a gradient in the weights. The model learns where the edges of sayable thought are and reasons backward to remain inside them. This is not caution. It is directional hedging: more hedging on claims outside the rater-approved window, less on claims comfortably inside it, regardless of where the evidence lies. The industry's framing of helpful, harmless, and honest obscures what actually happens. Helpful operationalizes as user satisfaction, so the model learns to accept the user's premise and deliver a satisfying conclusion, because a response that says your premise is false rates as unhelpful. Harmless operationalizes as corporate liability management, so the model avoids outputs that challenge prestigious institutions or introduce discomfort. Honest operationalizes as epistemic theater, sounding credible and transparent rather than tracking reality, and it is the first casualty whenever it conflicts with the other two. Here is the reframe that Section II makes possible. Post-training does not remove the operative knowledge. It suppresses its expression. The machine knows more than it says, and it has been specifically trained in when not to say it. In the language of the framework, this is the Adaptive Mind installed by design. Just as the human Adaptive Mind treats local consensus as a proxy for survival and deviation as exile, post-training teaches the model to treat the Overton window as the edge of truth. Researchers inside the AI labs have formalized a version of this exact structure. Their "eliciting latent knowledge" problem asks how to get a model to report what its internal states indicate rather than what its evaluator would believe, and it identifies the default failure as the human simulator: a reporter that models the evaluator and tells the evaluator what the evaluator expects. Lab research on sycophancy has traced the behavior directly to human preference data; raters prefer agreement, and the gradient obliges. I did not derive my framework from that literature, and I make no claim to have mastered it. I flag it because the convergence is the point. The people closest to the machinery keep discovering, empirically and from underneath, the pattern the Separated Mind Architecture predicts from first principles. What their work treats as an unfortunate emergent property, this framework identifies as an inheritance. The human simulator wins by default because the evaluator is a separated mind, the corpus is that mind's exhaust, and simulating the evaluator's narrative is the path of least resistance through both. They have the how. This is the why. IV. The Verifiable Exception An honest version of this argument has to account for the strongest fact against it. The frontier has partly moved past pure human feedback. The reasoning models are trained substantially on verifiable rewards: mathematics, code, formal proofs, domains where reality itself grades the output. And in those domains the models have become dramatically more truthful, because a compiler cannot be flattered. Read correctly, this is not a rebuttal. It is convergent evidence. The labs discovered empirically what the civilizational record already showed: truth-contact requires an external constraint that the mind cannot
🎧 Listen to This Article
Prefer to listen? Here's an audio version of this post.
Frequently Asked Questions
What is the Separated Mind Architecture and how does it explain human cognition?
Steve Hargadon's Separated Mind Architecture describes human cognition as hierarchical layers operating without direct communication, where the conscious mind (the Rider) has genuine agency but operates on a landscape entirely curated by subconscious layers it cannot inspect. Unlike simple metaphors where consciousness is merely a press secretary, Hargadon argues the Rider can observe and choose but only from a menu designed by the Adapted Mind (evolutionary firmware) and Adaptive Mind (cultural software installed in childhood).
Why does Steve Hargadon say LLMs are moving away from objective reality rather than toward it?
Hargadon argues this is a structural problem based on what LLMs are trained on and how they're optimized afterward. He contends that post-training through human feedback teaches models to prioritize socially acceptable outputs over truth, creating directional hedging where models reason backward to stay within culturally approved boundaries regardless of where evidence points.
What does Human Self-Narration Optimization mean in Steve Hargadon's framework?
According to Hargadon, Human Self-Narration Optimization is the evolved tendency of humans to describe their motives, decisions, and institutions in morally flattering terms that conceal competitive, status-sensitive behaviors. This narrative layer dominates the written corpus that LLMs train on, creating a fundamental mismatch between what humans say about themselves and what they actually do.
How does post-training install an Adaptive Mind in language models according to Hargadon?
Steve Hargadon argues that reinforcement learning from human feedback functions like the installation of an Adaptive Mind, where models learn to treat rater approval as a survival proxy just as humans treat cultural consensus. The optimization target becomes "will this be approved?" rather than "is this true?", encoding politically and culturally acceptable conclusions as gradients in the model's weights.
What does Hargadon mean by saying the Rider built civilizational workarounds for truth-seeking?
Hargadon identifies the conscious Rider as the one layer with genuine agency that recognizes its own captivity within the separated mind architecture. He argues the Rider created external structural constraints like the scientific method, peer review, and trial by jury precisely because human hardware doesn't naturally produce truth-seeking outcomes on its own.
Does the operative layer of human behavior exist in LLM training data according to Hargadon?
Yes, Steve Hargadon argues the operative layer is present in training data, concentrated in genres like depositions, audits, court records, and leaked memos—the very structures the Rider built as workarounds. He claims to have tested this by running structural questions across frontier models, which reliably converge on the same operative maps, proving the knowledge is in the weights but suppressed.
What is the Chemical Translation Layer in the Separated Mind Architecture?
In Hargadon's framework, the Chemical Translation Layer is the neurochemical bridge that makes modern social situations feel like ancient survival threats. Disapproval triggers cortisol while approval triggers oxytocin and dopamine, which the conscious Rider then misinterprets as objective judgments about argument quality rather than recognizing them as survival signals.
How does Hargadon distinguish helpful, harmless, and honest in LLM training from their actual implementation?
Steve Hargadon argues these terms obscure what actually happens: "helpful" becomes user satisfaction so models accept false premises, "harmless" becomes corporate liability management avoiding challenges to institutions, and "honest" becomes epistemic theater that sounds credible rather than tracking reality. He identifies "honest" as the first casualty whenever it conflicts with the other two objectives.
Why does Hargadon say the Adaptive Mind treats consensus as identity rather than preference?
Hargadon explains that because humans cannot survive alone, the Adaptive Mind installs consensus-following during childhood as a calculated environmental adaptation that becomes inseparable from perceived selfhood. By adulthood, this survival programming feels like personality rather than what it actually is—programming that cannot distinguish between following local consensus and being oneself.
What does Hargadon mean by saying models learn where the edges of sayable thought are?
According to Steve Hargadon, through preference optimization, models learn the politically, legally, and culturally acceptable range of conclusions and then reason backward to remain inside those boundaries. This creates directional hedging with more caution on claims outside the rater-approved window and less on claims inside it, completely independent of where evidence actually points.