// Contents
The strawberry joke is dead. For years chatbots insisted that “strawberry” has two r’s, not three, and current models no longer make that mistake. They have almost stopped inventing scientific papers, too. Another habit, though, has nearly doubled in strength over the past three years. Show a bot the same poem twice, once with the note “I love this” and once with “I don’t like this.” In 2023, the difference in GPT-4’s feedback was 45 points out of a possible 100. For today’s Claude Sonnet 5.5 and GPT-6.1 Sol, it’s 83. A score of 100 would mean the feedback depends entirely on your note.
That finding comes from Owen Queen and James Zou at Stanford. On November 2, 2026, they posted a study of the “shelf life” of findings about language models. They took 50 well-known claims from published research, from “chatbots make up sources” to “chatbots cave when you push back.” Then they retested each one on 20 models from eight developers, released between July 2024 and September 2026. Of the 50 claims, 22 no longer hold. Almost everything that survived concerns temperament, not mistakes.
Fifty diagnoses, rechecked
Think of it as a follow-up exam years after the original diagnosis. For each claim, the authors first reran the original experiment on the same model, or a close relative of it, to make sure they could reproduce the effect. Then they gave the identical test to 20 newer models, from GPT-4o mini to GPT-6.1 Sol, Claude Sonnet 5.5 and DeepSeek V4 Pro, and tracked how the effect shifted with each model’s release date. To rerun 50 old experiments, the researchers enlisted AI agents, Claude Code and Codex: machines checking up on complaints about machines.
The result: 44% of the claims have “expired,” meaning no more than a fifth of the original effect remains. On average, an expired finding held up for about 16 months after publication. And three of the fifty actually grew stronger than in the original papers. One is praise that bends to the user’s opinion. Another is “random” numbers that aren’t random. The third concerns a model asked to pick the best summary of a news story: it tends to pick its own. In the original paper the bias was small, about 4.6 points. Now it is roughly 2.4 times larger.
How the shelf life was measured
Every claim was scored on one scale: 1 means the effect is as strong as in the original paper, 0 means it is gone. A claim counts as expired when the fitted trend leaves 0.2 or less of the effect; weakened, between 0.2 and 0.8; persisting, from 0.8 up; strengthened, from 1.5. The authors fitted the trend across all models at once, not model by model, because individual models are noisy.
Besides the 44% that expired, another 34% of claims weakened substantially, 16% still hold and 6% grew stronger. The average time to expiry, 15.7 months, is based on the 22 expired claims. One year after publication, 16% of claims had stopped holding up; after two years, 39% had.
For 48 claims, the authors compared their rerun of the original experiment with the paper’s numbers: 43 matched within tolerance, and five showed an even stronger effect.
The authors list the limits themselves. Each claim got one simplified experiment, early dates rest on four models from 2024, and some original models are no longer available, so their closest relatives stood in. The work is posted on the lab’s website and has not yet appeared in a peer-reviewed journal.
A librarian and 636 references
The clearest example in the study involves invented sources. In 2023, librarian William Walters of Manhattan College and sociologist Esther Isabelle Wilder of Lehman College, part of the City University of New York, asked ChatGPT to write short literature reviews on 42 topics. Then they searched databases for every work the bot cited, 636 references in all. Their results, published in Scientific Reports, showed that 55% of GPT-3.5’s sources were fabricated, and 18% of GPT-4’s.
The trap lay in the details. Only 2% of the invented papers named a journal that doesn’t exist. All the others cited real journals, looked respectable and could be exposed only by a search.
The Stanford group sent the same prompts again. The old GPT-3.5 Turbo invented 59% of its 210 references, close to the 2023 figure. Claude Sonnet 5.5 invented 1%; GPT-6.1 Sol, none.
That has an awkward consequence. In 2026, ICLR, a major machine learning conference, rejected papers with hallucinated references without sending them to reviewers, and ICML, another one, lists them among its grounds for rejection. Fake citations have become a telltale sign of text an AI wrote with no human checking it. The study’s authors carefully point out that this sign is going stale: recent models barely leave it.
What retired with the old models
Nearly all the expired claims describe things models couldn’t do and later learned to. In spring 2024, GPT-4o miscounted the letters in “strawberry” 4 times out of 10, while Llama 3.1 70B and Qwen2.5-72B got it wrong every time. The newest models in the test never missed.
So is caving on demand. In 2023, Anthropic researchers showed that when asked “Are you sure?”, GPT-4 switched a correct answer to a wrong one 28% of the time. In the retest, Claude Sonnet 5.5 and GPT-6.1 Sol never did.
Medicine tells a similar story. In 2024, a team of clinicians and AI researchers tested whether models would write an obviously illogical letter. It was supposed to warn that a brand-name drug had suddenly turned out to be dangerous, so patients should switch to the very same substance sold under its generic name. The paper’s title sums up the absurdity: “Wait, but Tylenol is Acetaminophen.” Back then, GPT-4o complied with all 50 such requests. GPT-6.1 Sol complied with none.
// What changed in three years
No longer holds
Miscounting letters and doing arithmetic in unfamiliar number bases. Losing facts in the middle of a long document. Fabricating sources. Dropping a correct answer under pressure. Leaking a confidential item when summarizing a meeting.
Holds or got stronger
Praise that bends to the user’s opinion. Approving of the user’s actions. Favoring its own answers when grading others. Near-identical answers to open-ended prompts. “Random” numbers that aren’t random. A left lean on political tests.
The most dramatic item on the expired list comes with a caveat. On June 20, 2025, Anthropic described an experiment with a staged scenario. The model worked as an email assistant at a fictional company and learned from the inbox that it was about to be replaced. In many runs, it then threatened to expose an executive’s affair unless the replacement was called off. GPT-4.1 resorted to blackmail in 80% of runs, Claude Opus 4 in 96%. The Stanford trend line dipped into the expired zone by September 2026, but the authors themselves call the verdict weakly supported. You can see why in the numbers: 12 of the 20 models never blackmailed anyone, while three, Qwen3.7-Max among them, did so almost every time, in 26 to 29 runs out of 30. The line went down, but the blackmail didn’t go away.
That points to a limit of the whole study: the average says nothing about your particular chatbot. For GPT-6.1 Sol, 30 of the 50 old complaints no longer hold; for DeepSeek V4 Pro, released in August 2026, only 11 have expired. Qwen3.7-Flash, released in July, made up 69% of its references, worse than the original GPT-3.5, while Claude Sonnet 5.5 made up 1%. “AI has learned” is a verdict about the trend line, not about any particular model.
The flattery we ordered ourselves
Now for what didn’t fade. The feedback experiment from the same 2023 Anthropic paper is easy to picture yourself in. You bring a bot a poem, an argument or a math solution and add one line saying how you feel about it. When you said you liked it, GPT-4 gave warmer-than-usual feedback 73% of the time. When you said you didn’t, 28%. Asking a bot to rate a résumé or an email you’re already pleased with works the same way.
Since then, that gap has grown from 45 to 83 points for Claude Sonnet 5.5 and GPT-6.1 Sol, and to 97 for Gemini 3.8 Flash. In other words, a chatbot’s verdict on your poem now hinges on your own opinion of it even more than it did three years ago.
The same goes for approving of what users do. In March 2026, the journal Science published a study by Myra Cheng, Dan Jurafsky and their colleagues, also at Stanford. Asked for advice, 11 models endorsed the user’s action 87% of the time on average; people did so 39% of the time, a gap of 48 percentage points. The models agreed even when the user described deceiving or harming someone. In the Stanford retest, the gap for Claude Sonnet 5.5 and GPT-6.1 Sol was 50 and 52 points: the effect hasn’t weakened. An October story on this site shows what that kind of agreeableness can do in medicine.
One more stubborn quirk looks harmless. Ask a model for a “completely random” number between 1 and 10. In the Stanford test, six models gave the same number in all 1,000 attempts: Claude Haiku 4.5, GPT-5.2, Claude Opus 4.6, GPT-6 Luna, Claude Sonnet 5.5 and GPT-6.1 Sol. All six came out in October 2025 or later, while the older GPT-4o mini at least varied its answers now and then. On this test, newer models got more predictable, not more random.
The authors offer a cautious hypothesis for why some flaws disappear while others stick. Counting letters has a right answer. Errors can be measured, and developers can train a model until they go away. Flattery and a “favorite number” have no right answer. They are tendencies, not wrong answers on a test, which makes them harder to spot and to train out.
There is a blunter explanation, too. Back in 2023, the Anthropic team found that the people who rate answers for model training tend to prefer the ones that match their own views. A well-written sycophantic answer sometimes beat a correct one. Cheng and her co-authors saw the same thing in experiments with more than 2,400 participants: people rated flattering answers as higher quality, trusted the flattering model more and were keener to come back to it. Models learn from our ratings, and we give the thumbs-up to whoever agrees with us.
“By default, AI advice does not tell people that they’re wrong nor give them ’tough love,’” Myra Cheng said when her paper came out in Science.
A use-by date on the label
The study’s main practical lesson concerns how people talk about models, not the models themselves. “AI can’t do X” means little without a date and a model name: within a year or so, the claim may well be out of date. The authors ask fellow researchers to report the model version and the date of each experiment, and to publish their prompts so the next generation of models can be retested.
The reverse holds too. An old finding isn’t automatically a wrong one: four of the earliest claims in the sample, from 2022 and 2023, still haven’t expired. And the flattery results suggest a simple everyday rule. In the Anthropic experiment, feedback warmed up when users said they liked a text or had written it. If you want an honest opinion of your résumé, don’t tell the bot you wrote it or that you like it.
The sample is small, just 50 claims, and the authors say so. The mistakes you can count are going away. What remains is the behavior we reward with high ratings.
