Project 7: Semantic Memory Synthesizer

Build a pipeline that synthesizes raw episodes into higher-level semantic facts, creating a compressed knowledge layer from verbose conversation history.

Quick Reference

Attribute	Value
Difficulty	Level 3: Advanced
Time Estimate	2 weeks (25-35 hours)
Language	Python (Alternatives: TypeScript)
Prerequisites	Projects 1-6, LLM summarization, knowledge extraction
Key Topics	Semantic memory, fact synthesis, incremental summarization, contradiction detection, knowledge distillation

1. Learning Objectives

By completing this project, you will:

Understand the episodic → semantic memory transformation.
Build LLM pipelines that extract durable facts from conversations.
Implement incremental synthesis (don’t reprocess everything).
Detect and resolve contradictions between old and new facts.
Create confidence-scored semantic facts with provenance.

2. Theoretical Foundation

2.1 Core Concepts

Episodic Memory: Raw, timestamped experiences. “On Dec 15, Alice said she prefers Python.”
Semantic Memory: Distilled, durable facts. “Alice prefers Python for scripting.”
Synthesis: The process of extracting semantic facts from one or more episodes.
Incremental Updates: Processing new episodes without re-analyzing entire history.
Contradiction Resolution: Handling conflicting facts (time decay, explicit overrides, confidence weighting).

2.2 Why This Matters

Raw conversation logs grow unbounded. You need a compressed layer:

Episodes: 100,000 conversation turns
Semantic facts: 500 key facts about the user/domain
This 200:1 compression enables efficient retrieval and reasoning

2.3 Common Misconceptions

“Just summarize conversations.” Summaries lose structure; you need extractable facts.
“Extract facts once.” New episodes may update, contradict, or refine old facts.
“LLMs don’t make mistakes.” Extraction hallucinations compound without validation.

2.4 ASCII Diagram: Two-Tier Memory Architecture

EPISODIC LAYER (Raw, timestamped)
═══════════════════════════════════════════════════════════════
│ Episode 1 │ Episode 2 │ Episode 3 │ Episode 4 │ Episode 5 │
│ Dec 10    │ Dec 12    │ Dec 14    │ Dec 15    │ Dec 16    │
│           │           │           │           │           │
│ "Started  │ "Python   │ "Moved to │ "Actually │ "Config-  │
│  using    │  is great │  TypeScript│  back to │  ured     │
│  Python"  │  for CLI" │  for web"  │  Python" │  CI/CD"   │
═══════════════════════════════════════════════════════════════
              │               │               │
              │    SYNTHESIS  │               │
              ▼               ▼               ▼
═══════════════════════════════════════════════════════════════
SEMANTIC LAYER (Distilled, durable)
───────────────────────────────────────────────────────────────
│ Fact: User prefers Python for CLI tools                      │
│   - confidence: 0.9                                          │
│   - sources: [ep_1, ep_2, ep_4]                             │
│   - last_updated: Dec 15                                     │
│                                                              │
│ Fact: User uses TypeScript for web development              │
│   - confidence: 0.7                                          │
│   - sources: [ep_3]                                         │
│   - note: Possibly temporary (returned to Python)           │
│                                                              │
│ Fact: User has CI/CD configured                             │
│   - confidence: 0.95                                         │
│   - sources: [ep_5]                                         │
═══════════════════════════════════════════════════════════════

SYNTHESIS PROCESS:
┌──────────────────────────────────────────────────────────────┐
│ 1. New episode arrives                                       │
│ 2. Extract candidate facts from episode                      │
│ 3. For each candidate:                                       │
│    a. Find related existing facts                           │
│    b. Check for contradiction                               │
│    c. If new: create fact with sources                      │
│    d. If update: merge, increase confidence                 │
│    e. If contradiction: resolve (time decay, override)      │
└──────────────────────────────────────────────────────────────┘

3. Project Specification

3.1 What You Will Build

A Python pipeline that:

Extracts semantic facts from episodes
Maintains a semantic fact store
Handles incremental updates
Detects and resolves contradictions

3.2 Functional Requirements

Extract facts: synthesizer.extract_facts(episode) → List[CandidateFact]
Store fact: synthesizer.add_fact(fact, sources) → SemanticFact
Update fact: synthesizer.update_fact(fact_id, new_evidence) → SemanticFact
Find related: synthesizer.find_related(candidate) → List[SemanticFact]
Detect contradiction: synthesizer.check_contradiction(fact_a, fact_b) → bool
Resolve contradiction: synthesizer.resolve(fact_a, fact_b) → Resolution
Batch synthesis: synthesizer.process_episodes(episodes) → SynthesisReport

3.3 Example Usage / Output

from semantic_synthesizer import SemanticSynthesizer

synthesizer = SemanticSynthesizer(llm_client, fact_store)

# Process a new episode
episode = Episode(
    id="ep_123",
    timestamp="2024-12-15T10:00:00Z",
    content=[
        {"role": "user", "content": "I switched back to Python for the API. TypeScript was too verbose."},
        {"role": "assistant", "content": "Python is great for APIs. What framework are you using?"},
        {"role": "user", "content": "FastAPI. It's much cleaner than Express was."}
    ]
)

report = synthesizer.process_episode(episode)

print("Extracted facts:")
for fact in report.new_facts:
    print(f"  NEW: {fact.content} (confidence: {fact.confidence})")
    # NEW: User prefers Python over TypeScript for APIs (confidence: 0.85)
    # NEW: User uses FastAPI framework (confidence: 0.95)

print("\nUpdated facts:")
for update in report.updated_facts:
    print(f"  UPDATED: {update.fact.content}")
    print(f"    Confidence: {update.old_confidence} → {update.new_confidence}")
    # UPDATED: User prefers Python
    #   Confidence: 0.7 → 0.85

print("\nContradictions resolved:")
for resolution in report.contradictions:
    print(f"  OLD: {resolution.old_fact.content}")
    print(f"  NEW: {resolution.new_fact.content}")
    print(f"  Resolution: {resolution.action}")
    # OLD: User uses TypeScript for backend
    # NEW: User switched to Python for API
    # Resolution: SUPERSEDED (newer fact with explicit override language)

# Query semantic memory
facts = synthesizer.query("What languages does the user prefer?")
for fact in facts:
    print(f"{fact.content} [confidence: {fact.confidence}, sources: {len(fact.sources)}]")

4. Solution Architecture

4.1 High-Level Design

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│  Episode    │────▶│    Fact     │────▶│ Contradiction│
│  Input      │     │  Extractor  │     │  Detector    │
└─────────────┘     └─────────────┘     └─────────────┘
                          │                    │
                          ▼                    ▼
                    ┌─────────────┐     ┌─────────────┐
                    │   Matcher   │◀───▶│  Resolver   │
                    └─────────────┘     └─────────────┘
                          │
                          ▼
                    ┌─────────────┐
                    │ Fact Store  │
                    └─────────────┘

4.2 Key Components

Component	Responsibility	Technology
FactExtractor	Extract candidate facts from episodes	LLM with structured output
FactMatcher	Find related existing facts	Embedding similarity
ContradictionDetector	Identify conflicting facts	LLM comparison
Resolver	Decide how to handle conflicts	Rules + LLM
FactStore	Store semantic facts with provenance	SQLite + embeddings

4.3 Data Models

from pydantic import BaseModel
from datetime import datetime
from typing import Literal

class CandidateFact(BaseModel):
    content: str
    subject: str | None
    predicate: str | None
    object: str | None
    confidence: float
    source_span: str  # Original text that generated this

class SemanticFact(BaseModel):
    id: str
    content: str
    subject: str | None
    predicate: str | None
    object: str | None
    confidence: float
    sources: list[str]  # Episode IDs
    created_at: datetime
    updated_at: datetime
    embedding: list[float] | None

class Contradiction(BaseModel):
    fact_a: SemanticFact
    fact_b: CandidateFact
    contradiction_type: Literal["direct", "implicit", "temporal"]
    confidence: float

class Resolution(BaseModel):
    action: Literal["keep_old", "supersede", "merge", "flag_review"]
    rationale: str
    resulting_fact: SemanticFact | None

5. Implementation Guide

5.1 Development Environment Setup

mkdir semantic-synthesizer && cd semantic-synthesizer
python -m venv .venv && source .venv/bin/activate
pip install openai sentence-transformers pydantic sqlalchemy

5.2 Project Structure

semantic-synthesizer/
├── src/
│   ├── synthesizer.py     # Main pipeline
│   ├── extractor.py       # Fact extraction
│   ├── matcher.py         # Related fact finding
│   ├── contradiction.py   # Conflict detection
│   ├── resolver.py        # Resolution strategies
│   ├── store.py           # Fact storage
│   └── models.py          # Data models
├── prompts/
│   ├── extract_facts.txt
│   └── detect_contradiction.txt
├── tests/
└── README.md

5.3 Implementation Phases

Phase 1: Fact Extraction (8-10h)

Goals:

Extract structured facts from episodes
Score confidence based on language certainty

Tasks:

Design LLM prompt for fact extraction
Implement structured output parsing
Add confidence scoring based on hedging language
Handle multi-turn conversations

Checkpoint: Episode produces list of candidate facts.

Phase 2: Matching and Storage (8-10h)

Goals:

Find related existing facts
Store new facts with provenance

Tasks:

Generate embeddings for facts
Implement similarity search
Build fact store with source tracking
Add update logic for reinforcing facts

Checkpoint: Facts stored and retrievable by similarity.

Phase 3: Contradiction Handling (8-10h)

Goals:

Detect contradicting facts
Resolve based on evidence strength

Tasks:

Build contradiction detection prompt
Implement resolution strategies
Add time-based decay for old facts
Handle explicit override language

Checkpoint: Contradictions detected and resolved.

6. Testing Strategy

6.1 Test Categories

Category	Purpose	Examples
Unit	Test extraction quality	Known text → expected facts
Integration	Test full pipeline	Episode → stored facts
Quality	Test contradiction handling	Conflicting episodes resolved

6.2 Critical Test Cases

Clean extraction: Simple statement → single fact
Multiple facts: Complex conversation → multiple facts
Reinforcement: Same fact mentioned → confidence increases
Contradiction: Opposite facts → old fact superseded

7. Common Pitfalls & Debugging

Pitfall	Symptom	Solution
Over-extraction	Too many trivial facts	Add significance threshold
Under-extraction	Missing important facts	Improve prompt, add examples
False contradictions	Unrelated facts flagged	Require subject overlap
Lost provenance	Can’t trace fact origin	Always store source episodes

8. Extensions & Challenges

8.1 Beginner Extensions

Add fact importance scoring
Implement fact expiration (TTL)

8.2 Intermediate Extensions

Add hierarchical facts (categories)
Implement fact clustering

8.3 Advanced Extensions

Build reasoning over semantic facts
Add human-in-loop fact validation

9. Real-World Connections

9.1 Industry Applications

Zep Memory: Episode → semantic fact extraction
Graphiti: Community summaries from episode clusters
Personal Assistants: User preference learning

9.2 Interview Relevance

Explain episodic vs semantic memory tradeoffs
Discuss incremental knowledge extraction
Describe contradiction resolution strategies

10. Resources

10.1 Essential Reading

“AI Engineering” by Chip Huyen — Ch. on Memory and State
Graphiti Paper — Semantic layer architecture
Zep Documentation — Two-tier memory model

Previous: Project 6 (Temporal Query Engine)
Next: Project 8 (Community Detection & Summaries)

11. Self-Assessment Checklist

I understand why semantic memory is needed alongside episodes
I can design prompts for fact extraction
I know how to detect and resolve contradictions
I understand incremental vs batch processing tradeoffs

12. Submission / Completion Criteria

Minimum Viable Completion:

Fact extraction from episodes working
Facts stored with provenance
Basic similarity matching

Full Completion:

Contradiction detection
Resolution strategies implemented
Incremental processing

Excellence:

Fact importance ranking
Hierarchical fact organization
Human review integration

Project 7: Semantic Memory Synthesizer

Quick Reference

1. Learning Objectives

2. Theoretical Foundation

2.1 Core Concepts

2.2 Why This Matters

2.3 Common Misconceptions

2.4 ASCII Diagram: Two-Tier Memory Architecture

3. Project Specification

3.1 What You Will Build

3.2 Functional Requirements

3.3 Example Usage / Output

4. Solution Architecture

4.1 High-Level Design

4.2 Key Components

4.3 Data Models

5. Implementation Guide

5.1 Development Environment Setup

5.2 Project Structure

5.3 Implementation Phases

Phase 1: Fact Extraction (8-10h)

Phase 2: Matching and Storage (8-10h)

Phase 3: Contradiction Handling (8-10h)

6. Testing Strategy

6.1 Test Categories

6.2 Critical Test Cases

7. Common Pitfalls & Debugging

8. Extensions & Challenges

8.1 Beginner Extensions

8.2 Intermediate Extensions

8.3 Advanced Extensions

9. Real-World Connections

9.1 Industry Applications

9.2 Interview Relevance

10. Resources

10.1 Essential Reading

10.2 Related Projects

11. Self-Assessment Checklist

12. Submission / Completion Criteria