AI Agent Memory Architecture: 4 Types on Postgres (2026)

AI agent memory architecture is the design of how an agent stores, updates and recalls what it learns — across steps, sessions and users — instead of forgetting everything the moment the context window closes. The blueprint most practitioners use has four layers: working, episodic, semantic and procedural memory. A crowded market of memory platforms would like you to believe this requires new infrastructure. It usually doesn't: all four layers map cleanly onto plain PostgreSQL with the pgvector extension — the same boring database you already run. That's exactly how I run agent memory in production, and this post walks through the full mapping, the actual SQL, and the honest cases where a framework does earn its keep.

Memory is not RAG

The most expensive mistake in agent builds right now is treating memory as "RAG with a different name." They differ on the one axis that matters:

• RAG is a stateless read path over a corpus you index. Its contents are a function of your documents. A thousand conversations later, the store is byte-for-byte identical.
• Memory is a stateful read-and-write path over what the agent itself has learned. Its contents are a function of the agent's history. Two agents with the same corpus but different histories answer differently.

The shorthand I use: RAG answers "what do the documents say?" — memory answers "what has this agent learned, and when did it stop being true?" That second clause is the part similarity search alone can't give you: facts expire. "Customer is on the Starter plan" stops being true the day they upgrade, and a vector index has no native notion of no longer true — your schema has to carry it.

The practical corollary: since the difference is the write path plus time — not the storage engine — the same Postgres instance happily serves both. Your RAG tables get rebuilt from source documents (mine are re-indexed by a script, like the docs pipeline I described in my RAG stack post); your memory tables get mutated by the agent inside transactions.

The four types of agent memory

The taxonomy everyone now uses — working, episodic, semantic, procedural — traces to the CoALA paper ("Cognitive Architectures for Language Agents", Sumers, Yao, Narasimhan & Griffiths, 2023), which borrowed it from fifty years of cognitive science: Tulving's episodic/semantic distinction from 1972, and the declarative/procedural split from classical architectures like Soar and ACT-R. Every commercial framework — Mem0, Zep, Letta, LangMem — speaks this vocabulary. Here is each type, and the Postgres construct that implements it.

1. Working memory — the scratchpad

The live thread: current messages, tool results, partial plans. It mutates on every step and gets compacted or discarded when the context window fills. In Postgres this is thread-scoped checkpoints:

CREATE TABLE thread_state (
  thread_id  uuid,
  step       int,
  state      jsonb,
  created_at timestamptz DEFAULT now(),
  PRIMARY KEY (thread_id, step)
);

If you're on LangGraph you don't even write this table — PostgresSaver checkpoints every graph step into your database, which is what gives you resume, replay and human-in-the-loop for free.

2. Episodic memory — the diary

What happened and when: every interaction, timestamped, append-only. Old episodes roll up into summaries so the log never has to be replayed in full:

CREATE TABLE events (
  id        bigserial PRIMARY KEY,
  thread_id uuid,
  user_id   uuid,
  role      text,
  content   text,
  embedding vector(512),
  ts        timestamptz DEFAULT now()
);
CREATE INDEX ON events USING hnsw (embedding vector_cosine_ops);

Retrieval is recency × similarity: a WHERE ts > … window combined with a pgvector nearest-neighbour search. A background job periodically summarizes closed episodes into a summaries table with its own embeddings.

3. Semantic memory — the facts

Distilled knowledge abstracted away from any particular conversation: preferences, entities, domain facts. The defining requirement is supersedence — new facts must be able to update or invalidate old ones, not just accumulate:

CREATE TABLE memories (
  id             uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id        uuid,
  namespace      text,
  content        text,
  embedding      vector(512),
  valid_from     timestamptz DEFAULT now(),
  invalidated_at timestamptz,
  superseded_by  uuid
);
CREATE INDEX ON memories USING hnsw (embedding vector_cosine_ops)
  WHERE invalidated_at IS NULL;

The write path: an LLM extracts candidate facts from the episode log, similarity-searches existing rows, then decides per fact — add it, update an existing one, invalidate a contradicted one, or do nothing. That's Mem0's published ADD/UPDATE/DELETE/NOOP loop, and it fits in a single transaction. The valid_from/invalidated_at pair is the plain-Postgres approximation of Zep's bitemporal model, where facts carry validity windows instead of being deleted.

4. Procedural memory — the habits

How to act: rules ("always confirm before touching production"), skills, and the agent's own evolving instructions. Updated rarely and deliberately, read at boot rather than per query:

CREATE TABLE instructions (
  agent_id   text,
  name       text,
  version    int,
  content    text,
  active     boolean DEFAULT false,
  updated_at timestamptz DEFAULT now(),
  PRIMARY KEY (agent_id, name, version)
);

The agent (or a background optimizer) writes a new version row and flips active — which gives you diffable, revertible prompt evolution. Letta's famous self-editing memory blocks are, under the hood, exactly this: rows in a Postgres database that the agent rewrites through tool calls.

The write path is the hard part

Notice that none of the storage above is hard. Four tables, two HNSW indexes, done. The genuinely hard 20% of agent memory is the write path: deciding what deserves remembering (extraction), resolving it against what's already known (dedupe and supersedence), compressing history without losing the load-bearing details (consolidation), and expiring what no longer matters (forgetting). Those are LLM-orchestration problems, not database problems — and they're what you are actually buying when you buy a memory framework. With Mem0's pgvector backend, the rows land in your own Postgres anyway; the product is the write-path prompts and orchestration around them.

A wild 30 days in agent memory

If it feels like memory suddenly got loud, it did. In the last month alone:

• June 4 — OpenAI shipped "Dreaming V3" for ChatGPT: a background process that synthesizes and revises memories between conversations — updating "going to Singapore in July" to "went in July" — instead of keeping an append-only saved-memories list. OpenAI's own factual-recall evals went from 41.5% (2024) to 82.8% with it.
• June 18 — Perplexity previewed "Brain": a self-improving memory for its Computer agent that builds a context graph of the agent's work (not the user) and refines it overnight.
• June 29 — Microsoft Research published Memora (with an ICML 2026 paper): store full memories, but retrieve through 6–8-word abstractions plus cue anchors. Microsoft reports up to 98% fewer context tokens than full-context inference, storing roughly half as many memory entries as Mem0 per conversation.
• June 30 — Harrison Chase published his "Wiki Memory" essay — not a product, a pattern: an agent-maintained wiki of files that compresses raw source data into a persistent, inspectable, versionable knowledge layer, with the synthesis precomputed rather than retrieved as raw chunks.
• July 10 — Mem0 published its "State of AI Agent Memory 2026" report, with new benchmark numbers for its April token-efficient algorithm and a census of 21 framework integrations.

One honest caveat before you quote any of those numbers: they are all vendor self-reported, on contested benchmarks. The LoCoMo methodology disputes between Mem0 and Zep have run in both directions — each has publicly re-scored the other downward. Read every memory benchmark as directional marketing until an independent harness settles it. If you want to evaluate memory for your own agent, measure it on your own traces (I use Ragas for exactly this).

The meta-point stands though: memory is having the moment context engineering had a year ago. It's becoming a first-class component of agent architecture — which is precisely why it's worth understanding what's underneath the platforms.

When a memory framework earns its keep

• Mem0 — when you want the extract/update/dedupe pipeline pre-built and maintained instead of writing the ADD/UPDATE/DELETE/NOOP prompts yourself. It's Apache-2.0, and pgvector is a first-class storage backend — so you're buying write-path orchestration, not a database.
• Zep / Graphiti — when temporal reasoning over relationships is the actual product requirement ("what did this customer believe before the plan change?"). That's a real differentiator — but note it runs on Neo4j, FalkorDB or Kuzu; Postgres support is an open feature request, so it's a second database to operate.
• Letta — when you want agents that autonomously manage their own memory (promote, consolidate, rewrite their persona) and you're adopting its whole agent runtime. Fittingly, a self-hosted Letta server persists everything in… PostgreSQL with pgvector. The thesis of this post, proven by the framework itself.
• Cognee — when "memory" for you really means an ontology-grounded knowledge graph built from a document corpus (the GraphRAG-shaped problem), not conversational state.
• LangGraph's PostgresSaver / PostgresStore — if you're on LangGraph, this isn't an extra framework at all. The official memory story of the reference agent framework is literally "point it at Postgres" — the Store even does pgvector-backed semantic search. LangMem adds pre-written extraction and consolidation logic on top, but it's still pre-1.0; I copy its patterns rather than depend on its API.

The rule of thumb I give people: if your requirement is "remember my users' preferences and recall them by similarity," a facts table plus one extraction prompt is a few hundred lines of code you can read, index, back up and debug with psql. Reach for a framework when the write-path orchestration is genuinely your bottleneck — not because a benchmark chart told you to.

What I run in production

My stack for a multi-tenant SaaS assistant: Postgres + pgvector, Voyage embeddings (voyage-3-lite, 512 dimensions), LangGraph for the agent loop, Ragas for evaluation. RAG tables and memory tables live in the same database with different write paths — the RAG side is rebuilt by an indexer, the memory side is mutated by the agent. Schema-per-tenant multi-tenancy means memory partitions naturally per customer, which quietly solves an entire class of "whose memory is this" problems.

Two practitioner notes on pgvector itself. First, HNSW is the right default for memory tables — better recall/latency than IVFFlat, no training step, and it tolerates the incremental inserts a write-heavy memory workload produces. Second, keep the extension patched: the 0.8.x line fixed filtered ANN queries with iterative index scans (0.8.0), patched a buffer-overflow CVE in parallel HNSW builds (0.8.2), and fixed HNSW-vacuum corruption cases (0.8.3–0.8.5, current as of July 2026). Memory tables get vacuumed hard by supersedence churn, so those vacuum fixes are not theoretical.

FAQ

What is AI agent memory architecture?

It's the layered design that lets an agent persist and recall information beyond a single context window: working memory for the live thread, episodic memory for past events, semantic memory for distilled facts, and procedural memory for learned rules and instructions. Each layer has different write patterns, lifetimes and retrieval strategies.

What is the difference between agent memory and RAG?

RAG is a read-only retrieval path over documents you indexed; its store never changes because a conversation happened. Memory is a read-write path over the agent's own experience — the agent extracts, updates, supersedes and forgets facts across sessions. If nothing writes to the store during a conversation, it's RAG, not memory.

How do AI agents remember previous conversations if LLMs are stateless?

The model itself remembers nothing — statefulness lives in the architecture around it. The agent persists events and extracted facts to a database during the conversation, then selectively loads the relevant slice (recent turns, similar facts, active instructions) into the context window on the next session. Memory is a retrieval discipline, not a model feature.

What are the four types of memory in AI agents?

Working (the live thread state), episodic (timestamped history of what happened), semantic (distilled facts and preferences), and procedural (rules, skills and the agent's own instructions). The taxonomy comes from the CoALA paper, which adapted it from cognitive science.

Do I need a dedicated vector database for agent memory, or is Postgres enough?

For the overwhelming majority of production agents, Postgres with pgvector is enough — HNSW indexes handle similarity search at memory-table scale easily, and you get transactions, backups and SQL observability for free. A dedicated vector database becomes interesting at hundreds of millions of vectors, which almost no memory workload reaches.

Should I use Mem0, Zep or Letta — or build my own agent memory?

Build the storage yourself — it's four tables. Then decide honestly whether you need a framework's write-path orchestration: Mem0 for pre-built extraction/dedupe, Zep for bitemporal knowledge graphs (with a graph database to run), Letta for self-managing agents. If you're already on LangGraph, its Postgres checkpointer and store cover working and semantic memory before you add anything.

Building an agent that needs to remember things — or trying to untangle memory from RAG in an existing product? That's the kind of system I design and ship. Get in touch.