Project 14: Memory Fabric Engine (Short/Long, Episodic/Semantic)

Build a serious memory architecture for assistants with memory tiers, lifecycle management, provenance, conflict resolution, and privacy-aware retention.

Quick Reference

Attribute Value
Difficulty Level 5: Master
Time Estimate 35-55 hours
Main Programming Language Python
Alternative Programming Languages TypeScript, Rust
Coolness Level Level 5: Pure Magic
Business Potential 4. The “Open Core” Infrastructure
Prerequisites Retrieval systems, data modeling, encryption basics
Key Topics episodic/semantic memory, embedding lifecycle, pruning, knowledge graph memory

1. Learning Objectives

  1. Separate short-term context from long-term durable memory.
  2. Implement episodic and semantic memory stores with clear semantics.
  3. Manage embedding model versions and backfills safely.
  4. Resolve memory conflicts with deterministic policies.
  5. Enforce privacy, deletion rights, and tenant/user boundaries.

2. Theoretical Foundation

2.1 Memory Tiering and Semantics

A reliable assistant memory is not one store. Working memory handles immediate conversation state. Episodic memory stores timestamped events. Semantic memory stores distilled facts. A graph layer captures relationships and provenance. This separation prevents noisy logs from polluting stable facts and allows you to answer both “what happened” and “what is true now”.

2.2 Lifecycle and Governance

Embeddings become stale when models change. Without versioning, retrieval quality degrades silently. Memory pruning must remove low-value artifacts while preserving auditability. Privacy controls must propagate to every persistence layer, not only the primary table.

3. Project Specification

3.1 What You Will Build

A memory service with:

  • tiered storage APIs
  • ingestion pipeline
  • distillation jobs
  • conflict resolver
  • retention/pruning scheduler
  • privacy deletion/export endpoints

3.2 Functional Requirements

  1. Ingest events into episodic store with provenance metadata.
  2. Distill facts into semantic store with confidence scores.
  3. Store entity links in a simple knowledge graph.
  4. Resolve contradictory facts deterministically.
  5. Support deletion, export, and consent status queries.

3.3 Non-Functional Requirements

  • Isolation: strict tenant/user namespace.
  • Traceability: every memory has source references.
  • Recoverability: replay log can reconstruct semantic state.

3.4 Real World Outcome

$ memoryctl ingest --user u-204 "I moved to Austin" --source chat
[Write] episodic mem_9001 confidence=0.88
[Distill] semantic fact_id=f_441 "current_city=Austin"
[Index] embeddings_v=2026.01 upsert=2

$ memoryctl query --user u-204 "where do i live"
[Conflict] old=Seattle new=Austin policy=recency+source_weight
[Answer] current_city=Austin (updated 2026-02-10)

4. Solution Architecture

4.1 High-Level Design

Input Event -> Episodic Log -> Distillation Worker -> Semantic Store
      |                                  |
      v                                  v
 Privacy Policy Engine            Graph Relations Store

4.2 Key Components

Component Responsibility Key Decisions
Episodic store append-only event history immutable logs
Semantic store active facts conflict policy with provenance
Graph store relationships edge confidence + source refs
Lifecycle worker re-embed, prune, merge versioned jobs

5. Implementation Guide

5.1 The Core Question You’re Answering

“How do I make memory useful, durable, and correct over months without violating privacy or trust?”

5.2 Concepts You Must Understand First

  1. Event sourcing basics
  2. Vector index lifecycle patterns
  3. Data retention and privacy rights
  4. Conflict resolution strategies

5.3 Questions to Guide Your Design

  1. Which memories are immutable events?
  2. Which memory facts are mutable beliefs?
  3. How do you backfill after embedding model upgrades?

5.4 Thinking Exercise

Given contradictory location statements over six months, define active truth rules and audit retention behavior.

5.5 The Interview Questions They’ll Ask

  1. Why split episodic and semantic memory?
  2. How do you migrate embedding models safely?
  3. How do you avoid cross-user retrieval leakage?
  4. What does “memory provenance” mean in practice?
  5. How do you implement compliant deletion with auditability?

5.6 Hints in Layers

Hint 1: Build append-only episodic log first.

Hint 2: Derive semantic facts asynchronously.

Hint 3: Tag all memories with source confidence.

Hint 4: Treat deletes as first-class workflows.

5.7 Books That Will Help

Topic Book Chapter
Data models “Designing Data-Intensive Applications” Ch. 2-4
Indexing trade-offs “Database Internals” indexing chapters
Retrieval systems “AI Engineering” retrieval chapters

5.8 Common Pitfalls and Debugging

Problem 1: deleted memories still retrieved

  • Why: vector store tombstones not propagated.
  • Fix: dual-delete + consistency checks.
  • Quick test: deletion should remove hits from all retrievers.

Problem 2: memory drift after re-embedding

  • Why: mixed embedding versions in one index.
  • Fix: version partitioning + staged migration.
  • Quick test: compare retrieval quality before/after migration on fixed benchmark set.

5.9 Definition of Done

  • Tiered memory model implemented
  • Conflict resolution deterministic and explainable
  • Embedding versioning and migration path exists
  • Privacy workflows pass integration checks