Karpathy's LLM Knowledge Base Architecture Changes How We Think About AI Agent Memory
Most developers building AI agents spend enormous energy on retrieval — chunking documents, tuning embeddings, wrangling vector databases — and still end up with agents that feel like they’re searching through fog. Andrej Karpathy just posted a different idea on X, and it’s worth paying close attention.
What Karpathy Proposed
In a post on X, Karpathy described a system he’s been calling “LLM Knowledge Bases” — a structured, markdown-first approach to building personal knowledge repositories that LLM agents can read, write, and maintain over time.
The architecture has three distinct stages:
- Ingest: Raw source material — research papers, GitHub repos, articles, datasets, even images — gets dropped into a
raw/directory. Karpathy uses the Obsidian Web Clipper to convert web content into local.mdfiles, keeping images alongside the text so the LLM can reason over them with vision. - Compile: Instead of indexing files into a vector store, the LLM reads the raw material and writes a structured wiki — summaries, backlinks, encyclopedia-style articles, and concept maps all generated and organized by the model itself.
- Maintain: The LLM runs periodic “health checks” on the wiki — scanning for inconsistencies, filling gaps, surfacing new connections between ideas, and keeping the whole thing coherent as new material arrives.
At around 100 articles and 400,000 words, Karpathy notes the LLM can navigate the wiki via index files and summaries to answer complex, synthesized questions — no embedding lookup required.
Why This Matters for AI Agents
The conventional approach to giving agents long-term memory is RAG: chunk your documents, embed them, and retrieve the closest vectors at query time. It works, but it has real problems. Embeddings are opaque — you can’t inspect what the model “remembers” without reconstructing it. Retrieval quality degrades on nuanced or cross-document questions. And the whole pipeline is brittle to maintain.
Karpathy’s approach flips the model. Instead of storing raw content and retrieving at query time, the LLM does the synthesis upfront — compiling raw data into a human-readable, linkable wiki that both humans and agents can navigate. The knowledge base becomes an artifact you can inspect, edit, and audit. Every claim is traceable to a markdown file. There’s no black box.
This is a fundamentally different philosophy: organize first, retrieve later — versus the RAG way of store everything, organize at query time.
The Deeper Implication
What Karpathy is describing isn’t just a personal productivity hack. It’s a template for how autonomous agents could maintain persistent, growing knowledge about a domain. An agent that ingests papers, compiles a wiki, runs health checks, and answers questions from that wiki is effectively doing a version of what a human researcher does.
The system also creates a natural feedback loop. As the wiki grows, the agent’s answers get better. As the agent’s answers get better, it can surface better raw material to ingest. The compounding effect is significant.
This also pairs naturally with the autoresearch repo Karpathy released earlier this year — autonomous agents running research loops on their own. A knowledge base like this is exactly the kind of persistent memory layer that makes multi-session agentic work tractable.
What to Take Away
If you’re building agents and still reaching for a vector database as the default answer to “how does the agent remember things,” it’s worth asking whether a structured, LLM-compiled markdown wiki might serve your use case better. Especially for knowledge-heavy domains where auditability and human readability matter.
Karpathy has a way of pointing at things before they become obvious. This feels like one of those moments.
Further Reading
Ole Lehmann’s X thread breaking it down
AI Disclosure
This document is drafted by an AI skill and is provided for informational and governance support purposes only. It does not constitute legal advice or a formal compliance determination. Do not publish or rely on this notice as a substitute for review by qualified legal counsel or a licensed compliance professional with jurisdiction-specific expertise.