Why the privacy case against cloud AI memory isn't paranoia

A lawyer, a clinician, or an accountant thinking about building an AI knowledge base out of client files faces a question that sounds paranoid until you look at what has actually happened. If everything sensitive the practice holds gets fed to a cloud model to build and maintain that knowledge base, what happens to it if the vendor is ever compelled to hand it over?

Here is the answer, with the actual dates attached rather than a guess.

What actually happened

In NYT v. OpenAI, a court order issued in May 2025 required OpenAI to preserve output logs it would otherwise have deleted, including chats users had explicitly deleted themselves. The blanket order ended on 26 September 2025, but the corpus of data covered by it between April and September 2025 remains under legal hold. Whatever a vendor’s own deletion policy says, it yields to a court order covering data the vendor holds. That is not a comment on OpenAI specifically; it is a demonstration of what “we delete your data” means once litigation is in the room.

Retention terms shift too, and not always toward more privacy. Anthropic’s consumer terms changed in August and September 2025, moving consumer Claude plans to opt-out training with five-year retention for users who allow it. Commercial and API terms were excluded from that change, which matters, but it is a reminder that a policy commitment made when you signed up is not a guarantee about what applies a year later. Zero-retention arrangements exist and are real, but they are enterprise-gated contract terms, not something built into the architecture by default.

The honest complication

Here is the part that does not fit a simple “always go local” answer. The best independent evaluation of local models found in this field (a 30-day, six-stack study published in May 2026) puts a real floor under what local, private AI can currently do: reliable tool-calling starts around 27 to 32 billion parameters, and a supervised local agent handles capture, summarisation, and note updates about as well as a frontier cloud model on short tasks. But long-horizon work, past roughly five to eight steps, and anything requiring real judgement in curating what goes into the trusted layer, is still frontier-model territory. Revealed preference bears this out: most people actually running these knowledge bases in 2026 still route through frontier cloud models, because capability currently outbids privacy for most users, most of the time.

That is not an argument for ignoring the privacy question. It is an argument for being precise about where it bites. The honest posture that follows from the evidence is hybrid: local models for the continuous, private capture of the most sensitive material, frontier cloud models for the curation and judgement work, and the most sensitive files excluded from cloud sessions altogether rather than trusted to a policy promise.

What this means for a practice holding client data

For a professional practice, the sensible questions are concrete, not abstract. What is actually going into the AI’s context on a given session? If it is a client’s file note, a matter summary, a clinical record, does it need to leave the practice’s own systems at all to get the value you are after? Where a cloud model is genuinely the right tool, are secrets and the most sensitive identifiers kept out of what gets sent, as a matter of design rather than trust? And is there a documented answer, in advance, for what happens if a vendor is ever legally compelled to hand over data it holds, rather than finding out the hard way?

This is the same principle behind the architecture we built for ConfideAI, our product for mental health practitioners, which processes and de-identifies session content on the practitioner’s own device before anything reaches a cloud model at all. The privacy question for an AI knowledge base holding client data is not solved by a vendor’s promise. It is solved, when it needs to be solved properly, by deciding at the architecture level what never leaves the building.

None of this is legal advice, and it is not a substitute for advice from a practice’s own lawyer or professional body about what its specific confidentiality obligations require. It is a description of what has actually happened so far, offered so a practice can ask its adviser sharper questions.

Working out where the architecture line sits for a specific practice, given what it actually holds and what the AI is actually for, is exactly the kind of assessment we do in our AI strategy and governance work, and it is a live question for any professional services practice holding client-confidential material.