I’ve watched too many engineers hand an agent the keys to their entire knowledge base and act surprised when it sets the place on fire.

It happens the same way every time. Someone wires up a capable model with filesystem access, points it at a folder full of notes, code, contracts, or research, and turns it loose. For a while, it’s magic. Then one day the agent decides the cleanest way to “fix” a doc is to overwrite it. Or it confidently refactors a file it didn’t fully read. Or it does the right thing on paper but to the wrong copy of the file. The work that vanishes is rarely the work you’d choose to lose.

But the deeper problem is structural, and it’s not really about any single agent’s bad day. Three failure modes keep showing up across teams I talk to (Unfortunately, this is usually after they lost some data already).

Destructive writes. The one above. An agent with broad write access treats your source-of-truth folder as a workbench. Users point the agent to “things it can touch”, but define no real read/write boundary, and the agent doesn’t know which files are sacred because nothing told it, the context was compacted and instructions no longer there, or it simply gaslit itself into believing “rm ~/Pictures/Wedding“ is the most effective way to “organize my pictures please”.

Vendor lock-in. A team is convinced to adopt whatever “memory” feature their current AI product ships. Six months later they want to switch models, switch harnesses, or just take their context with them, and they can’t. The memory lives inside someone else’s product and was never theirs to leave with.

Black-box memory. Even when the agent does remember things, no one can audit what it remembers. You can’t diff it. You can’t roll it back. You can’t ask “what does the agent currently believe about the refund policy?” and get a straight answer. The memory is not an artifact, but more of an amorphous “vibe” that lives “somewhere”.

Each of these is, on its own, the kind of problem you can patch. Together, they’re a category error. We’ve been treating an agent’s memory like a feature of the agent. It isn’t. It’s a substrate the agent runs on top of, and right now, almost nobody owns theirs.

Why you should own your agent’s memory layer

If you take something from this article, let it be this: your agent’s memory layer should be a thing you can hold in your hand.

Local. Portable. Inspectable. Yours.

That’s a different shape than what most products are selling you. The closed-source harnesses currently competing to be your agent’s memory have a real interest in keeping your context inside their walls. Once your knowledge lives in their format, in their cloud, behind their account, you’re way less likely to churn. The pitch is “we’ll remember things for you.” The unspoken second half is “and we’d prefer you couldn’t take it with you.”

You don’t have to take that deal. And you don’t have to take my word for it, I’m not the only one saying this.

Agent Library

The Agent Library is the memory layer we wanted and didn’t have. It’s a local SQLite file with hybrid semantic + keyword search, ingests text, code, PDFs, and images, speaks MCP natively, and runs entirely on your own machine. No cloud account. No vendor seeing your documents. No re-embedding the same docs in five different tools.

We’re open-sourcing it today. The repo is at https://github.com/ArcadeAI/agent-library, and there’s a demo video that walks through what it actually feels like to use. There I show you:

  • How to setup a new Agent Library
  • How to add sources
  • How to keep the Library indexed
  • How to use it in multiple agent harnesses
  • How to use it in the CLI
  • How to search content in the library
  • How to manage the content inside

If you want all the details go watch it.

The read/write boundary

The single most important design decision in the Agent Library is that, from an agent’s point of view, the library is read-mostly.

Most of what an agent needs from a memory layer is read access: search the library, pull up the relevant chunk, ground the answer. The Agent Library’s MCP server exposes a small set of read tools (search_library, read_from_library, and friends) that cover that 90% case. It also exposes write tools (add_to_library, update_library_doc, remove_from_library) for the cases where you want the agent to maintain its own notes, but those are opt-in.

If you’re starting fresh, expose only the read tools. A library configured this way cannot, by construction, delete your notes, overwrite your code, or corrupt your PDFs. The agent can ask questions of your knowledge base all day; it cannot edit it.

When something does go wrong (and at some point it will) the second property kicks in: The Agent Library’s entire state is a single SQLite file you own. You can cp it before letting an agent loose. You can diff two versions of it. You can roll back to last Tuesday’s index. You can open it in any SQLite browser and see, byte for byte, what the agent currently knows. That’s a different relationship to “memory” than a black box in someone else’s cloud, and it’s the relationship you want the first time an agent surprises you.

To be perfectly clear, the Agent Library is a sharp tool. It narrows the blast radius, but does not eliminate it. If you give an agent shell access, filesystem write access, and Librarian in the same breath, Agent Library can’t save you. The agent can still rm -rf its way through your system no matter how careful the Agent Library is. The point is that the agent’s memory layer shouldn’t be the thing that betrays you, and right now, for too many people, it is.

Why this shape

The Agent Library is opinionated, and the opinions belong to our CTO, Sam Partee, who built it.

The local-first stance, the read/write split, the MCP-native surface, the choice of SQLite over a hosted vector database, the insistence that the index be a file you can hold. Those are all his calls, and they’re not arbitrary. They come from years of watching teams ship agents into production and discover, the hard way, which corners can be cut and which can’t. Memory is one of the corners that can’t.

That’s also why we’re releasing it as open source. Agent Library isn’t a part of Arcade’s product, but an agentic building block. The kind of thing we built for ourselves because we needed it, and that we think enough people need that it shouldn’t sit in a private repo.

Arcade’s agentic building blocks

Agent Library is the first of several pieces we’re going to open-source like this. Things we built for ourselves, with opinions baked in, that we think are useful on their own. Independent of whether you ever touch the rest of what we make.

If you’re building agents and you’ve felt any of the failure modes at the top of this post, grab it here, point it at a folder, and try it. Start with the read-only tools. Snapshot the SQLite file before anything ambitious. And if something blows up anyway tell us, or even better, open an issue or PR. That’s how the next building block gets sharper.

Your agent’s memory shouldn’t live in someone else’s product. Now it doesn’t have to.

FAQ

How is this different from other memory products?

Most “memory” products are trying to extract facts about you from your chat history (“user prefers Python”, “user’s company is Arcade”) and feed those facts back into future sessions. Agent Library doesn’t do that. It indexes a corpus you already have, notes, code, PDFs, images, and gives the agent a way to search it. The difference is where the memory lives. Your index is a single SQLite file on your disk. You can copy it, diff it, snapshot it, open it in any SQLite browser, move it between machines. The memory is yours in the literal sense.

Isn’t this just RAG?

The retrieval shape is RAG, hybrid semantic plus keyword search over chunked documents. But RAG is a technique, and Agent Library is a stance about where the technique should run. The contribution isn’t the embeddings or the chunking. It’s the boundary (a read-mostly tool surface that an agent can’t easily corrupt), the substrate (one file you can hold in your hand), and the interface (MCP-native, so any harness can use it without bespoke glue). You could rebuild the retrieval part of Agent Library in an afternoon. The point is everything around it.

What actually stays on my machine?

By default, everything. The index, the embeddings, the sections, and (of course) the documents that you already had locally. The default embedding provider is local sentence-transformers, no network calls. The one escape valve is the OpenAI-compatible embedding endpoint: if you configure one, that endpoint sees the text it’s embedding. We point ours at a local OpenAI-compatible server, which keeps the data on the box. If you point it at a hosted API, that’s a choice you’re making explicitly.

What’s the relationship to Arcade’s product?

Agent Library is built on arcade-mcp-server, Arcade’s open-source MCP framework, the same way any Python app is built on its libraries. It does not require an Arcade account, an API key, or any hosted service. The Arcade product is a separate thing. Agent Library runs standalone.

Is it production-ready?

It’s the memory layer we run locally ourselves, and the design decisions are settled, the read/write split, the SQLite substrate, the MCP surface aren’t going to churn. But it’s day-one open source. Expect rough edges, expect the API to tighten as people use it in ways we didn’t anticipate, and expect us to fix things faster if you tell us what broke. I’d start with the read-only tools and snapshot the SQLite file before anything ambitious.