The missing measurement in the AI second-brain boom
Every claim about AI knowledge bases saving time is self-reported. The one controlled experiment measured token economics, not benefit.
A client asks a reasonable question before signing off on any AI project: will this actually save time, and how much? For most AI work, you can point to a benchmark, a pilot, or at least a vendor’s own testing. For the AI “knowledge base” or “second brain” pattern, the honest answer in mid-2026 is that almost nobody has measured it at all.
The pattern is real and spreading fast. Andrej Karpathy’s gist of 4 April 2026 named it: a folder of markdown files an AI agent reads, writes, and maintains. It had passed 5,000 stars and forks within three months. Google formalised the same idea as a draft Open Knowledge Format on 12 June 2026. A hyperscaler moving to standardise a three-month-old pattern is a real signal. It is a signal of momentum, not of proof.
What the field actually has
The best longitudinal report published to date comes from Brian Buntz at R&D World (11 June 2026), one month into sustained use, maintaining roughly 760 pages under citation-required schema rules and lint gates. His verdict, from a disciplined practitioner who kept at it: “the time I spend maintaining the wiki and the time it saves me are roughly a wash.” The payoff, he found, concentrated on narrow topics the open web covers poorly. That is the field’s honest centre of gravity right now, not a failure, but not a win either.
The field’s one controlled experiment is more pointed. The archcheck project ran an A/B test in July 2026 on an agent-built wiki over a codebase, and found it saved nothing over letting the agent grep the source directly, because the wiki pages were larger than the files they described. Its author’s conclusion is the sharpest design principle the field has produced: token saving “appears only when a page aggregates facts scattered across many files (compression ratio > 1)”, and pages that mirror small source files “are pure maintenance debt”. Either the wiki earns its keep through compression, or it should not exist. That is one experiment, on one codebase, and it is still more than any comparable test of aggregate time saved across a general knowledge base.
A widely cited survey (Vibehackers, 4 March 2026) reports that most of 27 public Claude Code memory-system builds it looked at were abandoned within weeks. The title, date, and that finding are confirmed in the article’s own published metadata. Its methodology and the identity of the 27 builds could not be independently read at the time this paper was checked, so the claim is repeated here as the article’s own stated finding, not as an independently verified one.
What has not been measured, anywhere
Set those against the marketing. Jeremy Utley’s “Teammate Stack” (two Substack posts, 12 May and 16 June 2026) packages the same underlying pattern for a mass professional audience as eight persona files loaded into a chat assistant. The mechanics are sound. The claim attached to it, “11.5x smarter”, carries no cited methodology anywhere it is written. That is the pattern across the field: multiplier claims travel further than the measurement behind them, because there usually is no measurement behind them.
The research trail on this paper is blunt about it: no independent, controlled measurement of aggregate time saved by a markdown-based knowledge base exists anywhere, by any organisation, as of July 2026. The archcheck result measured the token economics of one narrow pattern. Buntz’s report is one practitioner’s honest self-tracking over one month. Everything else in circulation is a self-report or an unsourced multiplier.
What this means for an operator
None of this makes the underlying substrate a bad idea. A markdown knowledge base under version control, with a clear split between raw capture and reviewed knowledge, and an instruction file that encodes how the business wants the agent to behave, is cheap, portable, and inspectable regardless of whether it saves time this quarter. That part of the case does not depend on a multiplier.
What it does mean is that the time-saved claim is the one to verify yourself before it becomes the reason for the project. If a knowledge base is going to earn its keep, it will be through compression: pulling together facts that are genuinely scattered across places nobody has time to check, not through a wiki that mirrors what a search would already find. Pilot it, measure your own before-and-after on a real task, and treat “roughly a wash” as the honest floor rather than an outlier. Anyone selling a multiple without showing how they arrived at it is asking you to trust a number nobody in the field has actually produced yet.
That measurement discipline, applied before a business commits budget to an AI project rather than after, is the conversation we have with clients in our AI strategy and governance work.