matthitchman.com
← All posts

I built an Andrej Karpathy second brain that I can access from anywhere.

May 20268 min read

Everyone's talking about Andrej Karpathy's second brain, so I decided to build one. I wanted to access it from anywhere, using different LLMs.

The Karpathy pattern, briefly

In April, Karpathy published a GitHub gist describing what he calls an LLM Wiki. Worth a section on its own because it's the engine under everything else here.

The usual way to use an LLM with your notes is RAG: keep your raw notes, drop them in a vector database, search them at query time, hand the model the chunks it needs. That works. It's also expensive in tokens, and it re-derives the same conclusions every time you ask.

Karpathy's pattern flips it. Instead of searching raw notes at query time, you let the LLM maintain a wiki of structured, interlinked markdown files. When you add a new source, an article, a chat transcript, a workout log, the LLM doesn't just file it. It reads the source, updates the relevant pages, fixes cross-references, and notes where new information contradicts old claims. It touches ten or fifteen pages in one pass.

The wiki is what the model reads when you ask a question. Not the raw notes. The pre-organised, cross-linked, summarised version.

Two consequences make this powerful:

  1. The LLM keeps your notes alive. It re-reads what you already have, surfaces connections you didn't make, and updates summaries as new material lands. The bookkeeping that kills every human-maintained wiki, keeping cross-references current, summaries fresh, contradictions flagged, is exactly what LLMs are good at.
  2. You can ask it questions, properly. Not just "find me the note that mentions X" but "given everything I've logged about Y, what do I now think?". The model reads a clean, pre-built map of what you know, not the original pile. It's the difference between rummaging through a shoebox of receipts and asking your accountant for a P&L.

So I built one and called it HIKI (Hitchman's Wiki).

What I use it for

Three things, mostly.

Logging what I've done. Workouts, what I worked on today, things I've built, books I've finished. The kind of stuff I'd forget the details of by next week. It's sort of like keeping a journal, without having to actually keep one. And more useful, because an LLM can enhance it. Three months from now I want to remember that I've already solved this exact problem twice and forgotten how both times. HIKI keeps the receipts.

Capturing things I stumble across. A new AI tool, an idea I had in the car, a thread that struck me, a book recommendation a customer mentioned. These all used to die in the gap between "I'll remember that" and "what was that thing again". Now I tell the LLM to log it and move on.

Preserving the good bits of conversations with LLMs. This is the one I underestimated. I have long conversations with ChatGPT, Claude, whichever, about real things: a project I'm working on, a customer scenario, a half-formed argument. Most of those conversations are forgotten by the model the second I close the tab. Now, when one is worth keeping, I ask the LLM to summarise the conclusions to HIKI before we finish. Next time, the new conversation starts where the last one ended. With a different model if I want.

That last one is why this is worth building.

Without HIKI, an LLM helps me think for the length of one conversation. With it, the LLM picks up where I left off, with the conclusions I reached intact. It's the difference between a smart consultant you meet once and one who's been working with you for two years.

Log from anywhere

I wanted this to be frictionless. If logging takes more than ten seconds I won't do it. I'd already given up on three notes apps for that reason.

So the rule was: anywhere I have an LLM open, on any device, I can write to my brain in one sentence.

The way it works:

Any LLM with MCP support
    ↓
mcp.matthitchman.com  (Cloudflare Worker, OAuth gateway)
    ↓
hiki.matthitchman.com (Cloudflare Tunnel, no open ports on my router)
    ↓
Mac mini             (MCPVault, wraps the markdown folder)
    ↓
HIKI/                (iCloud-synced markdown, same folder Obsidian reads)

Four pieces, top to bottom:

The whole stack runs on hardware I already owned, costs me nothing a month, and stays up as long as the mini does.

Why markdown, not a database

The whole thing runs on flat markdown files. Not Postgres. Not a vector database. Not SQLite.

This felt wrong at first. Surely a real engineer would use a real database. But what I needed was a file an LLM can read, which is what a markdown file is. The model does the searching. The model does the filtering. The file system just has to be a file system.

Most "AI infrastructure" is solving problems you don't have yet. Start with a file.

What broke

Plenty:

None of these were particularly hard to fix, and probably took me longer to work out than they should have, as it was the first time I'd built an MCP.

What changed about working with an LLM

When I started building HIKI I thought of it as a place to store stuff. What I missed is that it changes how I work with the model.

Before HIKI, every new chat started with five minutes of context dump. Who I am, what I'm working on, that I'm based in NZ, that I code in Python at a beginner level. Now the LLM reads the relevant records files at the start and we skip all of that.

It's the difference between asking a smart stranger and asking a friend who already knows the backstory.

What's next

Honest accounting of the gaps:

None of these are hard. All of them are the kind of thing that gets pushed to "next weekend" forever.

In the meantime

HIKI works. I use it every day. It's the thing that took my relationship with LLMs from "useful tool" to "thing I'd notice if it went away".

Karpathy was right. Once the model remembers, it's a different tool.

If you build something similar, tell me. matt@matthitchman.com.

Matt