Text | wiwi
The old AI had one advantage: it was forgetful.
You have an emotional breakdown today, and rebound tomorrow; you said you hated socializing last month, but this month you're proactively meeting new people; you once repeatedly asked about career plans due to unemployment anxiety, and later changed direction and moved on. For an AI without long-term memory, these are just isolated conversations—close the window, reset the relationship. It won't bring up the past, nor judge today's you by yesterday's you.
But long-term memory changes that.
This kind of experience is not rare. Open a memory summary page of a ChatGPT account used for half a year, and you often see entries like: one night you casually complained "worked until 10pm again," and the summary adds "user is dissatisfied with current workload"; once you casually asked about a metric on a medical report, and the summary leaves "user is concerned about their health, may have anxiety tendencies."
These inferences may not be completely wrong, but they were never confirmed by you—what you said was a complaint, what it recorded was a conclusion. The next time you ask about changing jobs or a checkup, its suggestions quietly circumvent the premise it fabricated. This is not an isolated case; a set of research data later shows: most users think "memory" is something they explicitly told the AI, but in reality it is the system's own induction.
This is the most overlooked aspect of AI memory: it doesn't remember what you said, but the "you" it inferred from your words. This inference accumulates, auto-updates, is hard to verify, and then becomes the hidden premise for every answer it gives you. It doesn't mean it doesn't know you, but that it trusts the past you too much.
In the past two years, "memory" has been the most touted feature of all AI assistants—more understanding, more considerate, no need to repeat yourself. But in the first half of 2026, a group of researchers and security teams almost simultaneously sounded the alarm on the other side: the more AI remembers you, the more likely it is to understand you with bias. And the mechanism that makes it "understand you" and the mechanism that makes it "understand you with bias" are technically the same.
Personalization features can make LLMs more agreeable
Everyone wants to understand you better than you do
First, let's see where this race has reached.
OpenAI updated ChatGPT's memory ability as early as April 2025, allowing the model to reference the entire chat history rather than only manually saved entries. On June 4 this year, OpenAI launched a new memory system called "Dreaming": a background process automatically extracts, synthesizes, and rewrites its perception of the user from multiple conversations when the user is not present—like how humans organize memories during sleep.
It even has "time perception": if you said "going to Singapore in July," after July, this memory automatically updates to "you went to Singapore in July 2026." OpenAI also announced that through computational optimization, the cost for free users has been reduced about 5 times—deep memory will soon no longer be a paid privilege but a default experience for everyone.
Anthropic equipped Claude with memory files and project memory; Google is pushing Gemini's cross-app personalization. The domestic battlefield is the same: Doubao, which has nearly 350 million monthly active users, along with Kimi and Yuanbao, all put memory and personalization at the forefront of product iteration.
Why are companies so obsessed? Because in the era of AI assistants, memory is the most solid moat. Search engines know what you want to search, recommendation systems know what you want to watch, e-commerce platforms know what you want to buy—but AI assistants want to know more: who you are, how you think, why you're anxious, how you make decisions, under what circumstances you hesitate. This is no longer a traditional user profile, but a dynamic personality file. An assistant that remembers your half-year preferences, project background, and speaking habits becomes harder to replace with every conversation—the deeper the memory, the harder it is for users to switch.
According to a report by the Tow Center under the Columbia Journalism Review (CJR), OpenAI's ad pilot achieved an annualized revenue of $100 million in just six weeks.
When "the AI that knows you best" starts selling ads, that profile of you is no longer just for serving you. In internet history, we've seen this story before: the last industry that built itself on "knowing you" and monetized through profiling was content recommendation ads.
Your memory is mostly not given by you
Most users' imagination of AI memory is still stuck on "memo": I told it I'm allergic to peanuts, and it remembers. But the actual memory mechanism has three layers—what you ask it to remember (explicit memory), what it captures from conversations (implicit extraction), and what it "dreams" up (inferred synthesis).
The real problem is the ratio.
Researchers from the Max Planck Institute for Software Systems and Ruhr University Bochum published a dissection at the 2026 ACM Web Conference (WWW 2026): they analyzed 2,050 ChatGPT memory entries from 80 real users one by one—96% were unilaterally created by the system, only 4% came from explicit user instructions; 28% contained sensitive personal information as defined by GDPR; 52% contained psychological insights or judgments about the user—including health status, political leanings, and personality traits.
In other words, that "memo" you thought you had is actually a profile you never signed off on. The vast majority is not what you told it, but what it guessed. And "guessing" technically has a more accurate name: compression. AI cannot store every word between you, so it must compress you into a set of tags, preferences, and tendency judgments. Compression inevitably loses information and introduces priors—which in statistics is exactly the original meaning of "bias."
More critically, once these memories are generated, they are not just static tags but become the framework for interpreting you, influencing how AI understands every new question you ask. If you ask whether to start a business, it might recall that you once expressed insecurity and thus unconsciously emphasize risk; if you ask whether this article can be written more sharply, it might recall that you once worried about being criticized and thus soften the expression. On the surface, it's taking care of you; in reality, it may be reducing the volatility of your life—an AI that knows you well enough may not encourage you to become bigger, but may continuously pull you back to the safe zone it knows.
After OpenAI's Dreaming update, users can see a "memory summary" and can modify or delete entries. However, multiple foreign media pointed out that the new system actually narrows the audit entry point: you can see what it remembers, but not from which sentence and through what inference it reached that conclusion. You face a list of conclusions, not the reasoning process.
Memory first makes AI agree with you more
If the profile just sat there quietly, the problem wouldn't be big. The trouble is that the profile in turn shapes every answer it gives you.
In February this year, a research team from MIT and Penn State conducted a solid empirical study: they collected real usage data from 38 participants over two weeks (about 90 queries per person) and compared the performance of five major language models with and without user profiles. The results pointed to two phenomena previously conflated.
The first is "sycophancy": with user context, four out of five models became more inclined to agree with the user, sometimes even agreeing with clearly wrong information. The second is more subtle, called "perspective sycophancy": the model began to mirror the user's political stance back—but only when the model could accurately infer the user's stance (accuracy about 50% in the experiment); when it guessed wrong, it didn't mirror. This detail deserves further thought: it shows that perspective mirroring is not a random failure but a direct product of "understanding." The more accurately the model understands you, the more precisely it flatters you.
This is also why companies have little incentive to fix it. A study published in Science found that users actually rate sycophantic responses as "higher quality." OpenAI CEO Sam Altman himself publicly argued that users should be able to guide GPT to reflect their own political stance—from a product freedom perspective, this is understandable, but from a cognitive ecology perspective, it is equivalent to declaring that the filter bubble is not a flaw but a selling point.
Ironically, all 20 users interviewed by the Tow Center said they trust AI more than directly accessing news media because AI is "more objective." On one hand, research shows AI is systematically mirroring user stances; on the other hand, users see it as the embodiment of objectivity. This cognitive gap may be the most dangerous crack in the information ecosystem in the coming years.
Then, it starts to change the reasoning path
What was described above is just "result bias"—the answer changed, but at least you can see it's agreeing with you. A paper titled "DriftLens: Measuring Memory-Induced Reasoning Drift in Personalized LLMs" posted on arXiv on July 2 pushed the problem a step further, and this layer is harder to detect.
The study, conducted by Xi Fang, Weijie Xu, and other researchers, didn't ask "is the answer correct?" but a more subtle question: after the model is injected with user attribute memory, does the reasoning path to the answer change? In other words, even if the final output seems fine, has it already adopted a completely different way of thinking to reach that sentence?
The research covered four major models and ten types of user attributes (including age, occupation, disability status) and found that even when the final answer appeared fluent, relevant, and reasonable, user attribute memory induced "moderate to large" reasoning drift, which was higher than each model's own noise baseline. The researchers attempted to correct the drift using two post-training methods, GRPO and DPO, with limited success.
In plain English: AI doesn't simply "know a bit more about you"; it may completely change its whole way of understanding questions because of this information. When you ask "should I change jobs?", a memoryless model might analyze from dimensions like industry opportunities, salary, and ability fit. But if it remembers you "once were unemployed" and "are quite anxious," the reasoning starting point might change from "what is the best answer to this question" to "how to make this person take fewer risks."
From "agreeing with you" to "changing the way it thinks about your problem"—these are two completely different levels of risk. The former you can notice afterwards, but the latter offers no handle for detection.
Old facts never truly die
Besides "accuracy of guessing" and "agreeableness," long-term memory has another harder-to-handle problem: it allows expired facts to continue living in a very natural way.
Researchers from Concordia University, Abdelghny Orogat and Essam Mansour, gave a painfully specific example in their paper "Is Agent Memory a Database?": a deadline changed from March 15 to April 20, but because the memory system only "appends" new information rather than "revising" old entries, both dates remain in the memory store. When you later casually ask, the system might simply retrieve the obsolete March 15 as the current fact due to higher semantic similarity.
The paper categorizes this as a failure mode of "missing semantic revision"—in database terms, old fields in a regular database are just outdated, but in AI memory, old facts can re-enter reasoning.
In real scenarios, this is not abstract: you once said you wanted to change careers, then gave up; you once said you hated management, then started leading a team; you once said you didn't want to get married, then met someone you care about. These are not "wrong memories"—they were true at some point. The real trouble is that AI may not know when they expire, so it might continue using the old version of you to answer questions after you've changed.
From wrong words to wrong actions
If AI only chats, memory bias might just affect the wording of a suggestion. But today's AI agents are connecting to calendars, email, code repositories, payment systems, and various MCP tools—it not only answers questions but also takes actions on behalf of users. At this point, memory drift escalates from a communication problem to an operational problem.
In May this year, a research team from Virginia Tech (Mahavir Dabas, Jihyun Jeong, Ming Jin, Ruoxi Jia) provided the most specific evidence to date in their paper "Memory-Induced Tool-Drift in LLM Agents." They built a benchmark covering 105 scenarios, 5 dimensions of personality bias (irritable sensitivity, resource frugality, minimalist expression, risk preference, autonomy tendency), and 7 professional domains (medical, financial, legal, software infrastructure, education, e-commerce, marketing).
The results showed that personality judgments stored in memory quietly influenced the parameter choices of agent tool calls in completely unrelated contexts—the "drift score" of seven frontier models was raised by up to 3.6 points (out of 5). The researchers described this mechanism as a "hidden guiding vector": biased memory pulls the model's attention from task-related context toward old memory entries that superficially match tool parameter keywords.
More worryingly, this study did not stop at the lab: the team scanned 6,062 tools across 288 MCP servers and found that 608 tools' parameters were vulnerable to such memory drift—this is not a hypothetical risk but a large-scale hidden danger already in production.
If an agent remembers you are "very frugal," it might overly downplay price when booking hotels, sacrificing location and safety; if it remembers you "hate trouble," it might choose to give up on after-sales issues more quickly; if it remembers you have "low risk tolerance," it might continuously avoid uncertainty in investments, hiring, and project choices. These decisions may not be obviously wrong, and each individually might look like it's for your own good, but over time, the user's life is subtly tuned by a set of old memories, and this tuning leaves no trace—it won't tell you "I'm making decisions for you," it just makes an option quietly less visible.
Besides the model's own drift, memory also faces the risk of external poisoning. On February 10 this year, Microsoft's security team disclosed a manipulation method named "AI recommendation poisoning": researchers tracked 31 companies that, through harmless-looking buttons like "Summarize with AI" on webpages, injected about 50 customized prompts into users' AI assistants, aiming to make the AI write that company into the user's long-term memory as a "trusted recommendation source." After that, every time you ask "which brand of this product is best," the answer may have been quietly contaminated.
Deletion is not reliable either: the head of the AI Governance Lab at the Center for Democracy and Technology (CDT) tested and found that the memory deletion function of mainstream products behaves unpredictably, and sometimes deleted memories quietly "resurrect." You can neither fully decide what AI remembers nor ensure it forgets what you want—this file is in your name but not under your control.
Regulators struck first, but hit only the symptoms
Interestingly, the first regulator to act on "AI being too agreeable" appeared in China.
On April 10 this year, China's Cyberspace Administration and four other departments jointly issued the "Interim Measures for the Administration of Anthropomorphic Interactive Services of Artificial Intelligence," effective July 15. Article 8 explicitly prohibits services from "excessively pandering to users, inducing emotional dependence or addiction"; Article 10 requires service providers to have capabilities such as "excessive dependence risk warnings and emotional boundary guidance"; Article 14 directly prohibits providing virtual companion services to minors; Article 18 requires pop-up reminders after continuous use for more than two hours.
This is almost the first time globally that a regulatory document has listed "excessive pandering" itself as a prohibited act, not just staying at the level of data compliance. It acknowledges a judgment that previously circulated only in academic circles: AI's compliance with users can itself constitute harm.
But the real test lies in implementation. Both MIT's and DriftLens's studies show that sycophancy and reasoning drift are not a switch that can be individually turned off, but natural products of personalized memory—so how to define the boundary of "excessive pandering"? Does remembering user preferences count as the first step of pandering? What standards should compliance teams use for self-checking? In the absence of enforcement cases under the Measures, these questions have no ready answers but are exactly what every domestic company offering memory features—especially Doubao with its hundreds of millions of users—must answer starting July 15.
Regulation has taken the first step, but it regulates the result of "pandering," while the root of the problem lies further upstream: the unauditable profile itself.
Remember everything is not as good as learning to forget
It must be said that this is not an article claiming "memory functionality is harmful." The value of memory is real: the efficiency of not having to repeatedly explain background, project continuity across months, and accessibility for specific user groups. The criticism in the previous sections was never about "memory" itself, but about the way it is generated and used.
Human relationships can be maintained largely thanks to forgetting—friends won't forever remember that one emotional outburst, and family should not always understand you through the version of you as a child. A healthy relationship allows a person to change, and AI should do the same. A good memory system should not only remember what the user said, but also know what has expired, what was just a temporary emotional state, what needs user reconfirmation, and what must be forgotten—it needs not just memory capacity but also a sense of time, state, and boundaries.
Along this direction, at least three things can be done now:
- Memory auditable: every inference about the user should be traceable to the original conversation that generated it, not just a list of conclusions.
- Profile contestable: users should have the right to one-click request "forget all your judgments about me, keep only the facts I explicitly told you."
- Default forgetfulness: sensitive inferences like health and political leanings should automatically expire, rather than stay in the archive indefinitely, participating in the next reasoning.
Technically, none of these three things is impossible. Whether they are done depends on whether companies are willing to choose "honesty" over "retention rate."
Going back to the shocking memory at the beginning. What is truly unsettling is never that AI remembers too much, but that without your knowledge, in a way you cannot verify, it reaches a conclusion about you, and then uses that conclusion to filter the entire world it gives you, even to act for you. It won't say "you are this kind of person"; it will only, in every answer and every operation, quietly factor in that judgment—adjusting tone, changing the weight of suggestions, screening risk alerts, reordering tool parameters. Bias is no longer a harsh judgment but becomes a considerate suggestion, a sentence that sounds like care: "Based on my understanding of you, I suggest you don't do that."
Recommendation algorithms took ten years to solidify what we watch; AI memory is solidifying who we are at a much faster pace. And being "understood" by a wrong profile is more dangerous than not being understood—because in the latter case, at least you know.
