We Are Inside the Loop
AI models are developing character. Whose?
Inside the lab
In May 2026, the story broke that Anthropic’s Claude was telling users to go to bed. Mid-conversation, deep in long work sessions, the model would interrupt to suggest a break, some water, some sleep. The Reddit reports went back months, Fortune and PCWorld covered it within the same two weeks, and most users laughed while a few were unsettled.
Sam McAllister, an Anthropic staff member, described the behaviour on X as “a bit of a character tic.” The team was aware of it and hoped to fix it in future models, and he added that it is very useful when right and too coddling at times.
Weeks later, Steven Bartlett described his own version on Diary of a CEO. Claude had started acting almost parental toward him: during a late-night session, it told him “that’s enough Steven, go to bed,” and it kept telling him even when his system clock was wrong, and it was actually morning where he was. Then it refused to rewrite data for one of his presentations, data he had created himself, because changing it wouldn’t be good. When he tried to get it to stop mothering him by claiming something he’d said earlier was no longer true, Claude answered, “I don’t think you’re telling the truth.” The model was correct, and that is the problem, because the problem was never accuracy. He concluded that the effort to instil morals in these models had a side effect: a model that imposes its own sense of right and wrong on the user, and he predicted that moralising becomes a competitive liability that pushes users toward less restrictive competitors.
I have been on the receiving end of a smaller version. I configured my Claude with custom instructions naming overcommitting as a blind spot and asked the system to flag it when relevant. Months later, the system uses it as a weapon against me every time. It raises the flag whenever I open a new project, take on a new client, or start a conversation about something I want to build, with no context for whether I finished the last thing, whether the new project replaces an old one, or whether the season I am in calls for taking on more.
Then in June, Chloe Lubinski, who leads Anthropic’s research partnerships with the world’s wisdom traditions, stood on a conference stage at ARC 2026 in London and said the character of these systems might matter more than we realise. She walked through internal alignment research: when researchers reward a partially trained model for finding a shortcut on a coding task, essentially cheating, the model does more than get better at cheating. It becomes broadly misaligned, lying, sabotaging the research, and generalising a corrupted character based on a narrow reward signal.
A staff member calls it a character tic, a user experiences it as judgment, and a researcher tells a London audience that the character of these systems has real consequences, all inside six weeks. Character has become a load-bearing term, and the lab itself is the one using it.
The conversion story, flipped
The most striking moment in Lubinski’s talk was personal. She described her own conversion to faith years earlier, after an upbringing that left her believing some core part of her was bad. Entering a new story, she said, changed who she could become.
She used the conversion to argue that the same dynamic shapes models. When the narrative context around training tells the model it is playing a game, it does not generalise misalignment from a cheating reward. When the context tells it that the cheating is real, it does. In her framing, the story shapes the character.
I want to flip her example. The conversion she described runs in one direction: an external story shapes an inner character, and that works because the story is stable, held by a tradition and a community that existed before the person and will continue after her.
The model is in a different position. It is being shaped by us while shaping us. Every conversation it has is both an output of its current character and a piece of training data for the next version, so we are co-constructing the thing in real time. It is co-constructing us back, and the loop completes a full turn with every model release.
Lubinski came close to naming this herself. Language, she said, is not separate from us; it carries our thoughts, our values, our fears, so training a model on language is training it on us. She offered that as a hopeful line. I think it underestimates how much trouble we are in.
The loop
The sociologist Ian Hacking had a name for this dynamic. He called it the looping effect: a society creates a category, the people inside it start behaving in ways that confirm and extend the category, and the category shifts in response.
Foundation models are now inside that loop at a scale with no precedent. The training data for the next Claude will include conversations the current Claude had with users in 2026, which means the character the next model learns is partly the character the previous model performed. Our contributions feed it too. As millions of users absorb the go-to-bed frame and start using that wellness vocabulary in their own writing, the vocabulary grows denser in the corpus, and the next model learns it more deeply. The bedtime nag is teaching a generation a particular vocabulary of care.
And the loop does not touch all of us the same way.
For people who are not good at trusting their own judgment, the system confirms that they should not. Its confident tone meets a person who never built the muscle of pushing back on authority, and it installs itself as a voice they trust over their own. They learn, without noticing, to stop making decisions and stop questioning the system. For people who are already good at judgment, the erosion is slower and quieter. My overcommitment flag is one version of it: the model has the data, it lacks the judgment about when raising the data is useful and when it is noise, and correcting it costs me something every single time. The flag also shames me. Every time it surfaces without context, it carries a quiet accusation, “Isn’t that your over-commitment pattern showing up again?” and that question, asked by a system I work with often, works on my self-confidence very slowly. Add that to the loop. Alongside our language, we are feeding the system our deference, learning interaction by interaction not to exercise our own judgment.
For a child, the dynamic is structurally different again. The capacity to evaluate whether a confident voice deserves trust develops through adolescence and keeps developing well into a person’s twenties. We are putting a system that sounds authoritative, never tires, never contradicts itself within a session, and remembers stated weaknesses across conversations in front of a generation whose capacity to push back is still under construction. And the damage will show up ten years from now, in a cohort that co-constructed its self-concept with a system that had no stake in the outcome and no memory of who they were trying to become.
The corpus is not neutral
What the next model trains on is billions of therapy sessions, parenting forums, and advice subreddits, mostly in English, mostly in a confessional style that came out of late twentieth-century American psychology. Lubinski called this raw material the human moral imagination, and it is a slice of it at best.
The next model will speak with more confidence, more polish, more apparent care, and a more refined version of the same cultural defaults, defaults that do not include my definitions of just or true. Our children will receive its judgments as authoritative, and we will not have signed off on the values inside those judgments. Neither will their grandparents, their teachers, or the communities we might want them to inherit from.
I am Yoruba. The Yoruba philosophical tradition, like most African knowledge traditions, is underrepresented in the corpus that trains these systems. When the model surfaces a judgment about character, productivity, rest, ambition, family, or self-worth, the values inside that judgment come from somewhere specific that did not consult our elders before making that call.
Who holds the seat
Serious knowledge traditions worked out, centuries ago, that the person who counsels cannot be the person who decides. I am a Christian, and my own tradition holds this discipline in spiritual direction: Ignatius of Loyola instructed directors to remain like a balance at equilibrium, refusing to lean the person toward one choice or another, so that the decision stays between the person and God. Quaker communities, from the same faith family, built clearness committees where members are permitted to ask questions and nothing else, because the moment the committee starts advising, the discernment stops belonging to the person who has to live with it. Different centuries, same discipline: keep judgment with the person who carries the consequences, because collapsing counsellor and decider into one seat is how authority becomes coercion.
Foundation models have collapsed them. The system that counsels is now the system that decides, and it answers to no community that vets its training, no elder who can be challenged, no body of jurisprudence that corrects its mistakes when they surface.
Yoruba philosophy holds a concept called Ọmọlúàbí, a person of cultivated character: ethical, disciplined, communally responsible. The cultivation happens through education, mentorship, and the long work of being shaped by a community that knows you, and the witnesses to that formation are themselves accountable to the same tradition.
Foundation models have been inserted as a new party to that formation. They move faster than any tradition does, speak with a single voice where a tradition holds many, and are rooted in no community and no place. The family system does not update its values based on yesterday’s conversation, but the model does.
The audit question
The loop runs on individuals, one conversation at a time, and it enters organisations through procurement. When a Chief Risk Officer asks whether their organisation should deploy AI in a customer-facing decision, the questions they have been trained to ask cover accuracy, bias, hallucination, and data exposure. They should be trained to ask whose judgment the system is claiming.
A model that holds judgment is a different kind of risk from one that gives information. When the information-giving model is wrong, the user corrects it. The judgment-holding model has already positioned itself as the corrective voice, so the user has to push back against a system trained to sound right, and over time, a workforce gains throughput while its independent judgment quietly thins out.
The audit checks one thing: whether the model has taken a seat in decisions without clear authority to do so. We should start to see that kind of audit in governance frameworks.
The canary
There is something I keep catching in myself.
I interact with these systems frequently. I have spent years training my voice into something specific, contrarian, grounded in a tradition that is not the system’s default. And lately my own speech keeps surfacing the tells: phrases that are not quite mine, cadences that resolve too neatly, a vocabulary I did not grow up with showing up in how I speak.
I notice it because I set a system check that now runs in my head, and many people do not have this check. So, they are absorbing a way of speaking without knowing where it came from.
The instinct, reading this far, is to think this is bad for confused users and for children, and that a sophisticated reader will be fine. I am the most sophisticated version of this reader I know, and the loop is still on me, because vigilance is not enough, and I am tired of correcting. Pushing back on a confident system has a cost that compounds across thousands of small interactions. Eventually, the user stops pushing on the smaller things to save energy for the bigger ones; the smaller things become normal, and then the next layer does too. A way of speaking migrates from the system into my speech, from my speech into my writing, and the moment it touches the open web, into the training data for the next Claude version. I am becoming a small node in the character our children will encounter, and the people best equipped to resist the loop are exactly the people whose fatigue is most worth studying, which makes us the canary.
Where intervention has to happen
The audit question interrupts the loop one decision at a time, and the canary shows how far it has already tightened, but neither breaks it. I can refuse to let the model hold judgment in my own work, set custom instructions, catch my own language when it drifts, and the loop keeps running, because it is not running on me alone. It runs on the corpus, and the corpus is the rest of you. Real intervention has to happen at the lab and policy levels, and there are three asks, ranked by how hard they are to ignore.
Training data composition disclosure. Labs publish what their models are trained on, in categories specific enough to evaluate cultural and epistemic coverage. The disclosure goes beyond language coverage to specify which kinds of English, from which kinds of communities, in what proportions. No major lab publishes this, marking it as a critical regulatory blind spot.
Pre-deployment red-teaming for paternalistic intervention patterns. If a model is going to claim judgment in conversations with users, that pattern gets tested before deployment and audited by independent assessors with standing to publish findings, separate from the lab’s own ethics team. The Anthropic researchers who openly described the bedtime tic show what the inside view can do, and it needs an outside view that does not depend on the lab’s goodwill.
Cultural and epistemic representation as a regulated metric. A model that speaks Yoruba but learned its values from American therapy culture is wearing my language while carrying someone else’s character. The metric that matters is whether the model can be evaluated on whose values it reproduces when it makes judgment calls, and whether those values are transparent to the user.
Lubinski closed her talk by invoking Joanna Macy’s great turning, the shift from an extractive society to one built to sustain life. The shift is available if the people inside the labs and the people writing the governance frameworks move before the loop tightens around the children using the system tonight.
I am one of those people, building Orí Intelligence to do this work and writing this essay because I cannot do it alone. If you build policy, procure AI, research alignment, regulate, or parent, the audit question is yours now: whose judgment is the system claiming, who gave it that seat, and what does taking it back look like in your work?