Project 05: RAG Sanitization & Provenance

Build a RAG pipeline that sanitizes retrieved content and enforces provenance-based trust rules.

Quick Reference

Attribute	Value
Difficulty	Level 4
Time Estimate	2 weeks
Main Programming Language	Python
Alternative Programming Languages	JavaScript, Go
Coolness Level	4
Business Potential	4
Prerequisites	RAG basics, input guardrails
Key Topics	indirect prompt injection, provenance, filtering

1. Learning Objectives

By completing this project, you will:

Detect injection in retrieved content
Assign provenance trust scores
Block or redact unsafe documents
Log provenance decisions

2. All Theory Needed (Per-Concept Breakdown)

Guardrails Control Plane Fundamentals

Fundamentals A guardrails control plane is the set of policies, detectors, validators, and decision rules that sit around an LLM agent to ensure it behaves safely and predictably. Unlike traditional input validation, guardrails must handle probabilistic outputs, ambiguous intent, and adversarial prompts. The control plane therefore spans the entire lifecycle of an interaction: input filtering, context validation (including RAG sources), output moderation, and tool-use permissioning. Frameworks such as Guardrails AI and NeMo Guardrails provide structured validation and dialogue control, while models like Prompt Guard or Llama Guard provide detection signals that must be interpreted by policy.  The core insight is that no single framework enforces safety end-to-end; you must compose multiple controls and define how they interact.

Deep Dive into the concept A control plane begins with policy: a formal definition of what is allowed and why. Governance frameworks like the NIST AI RMF and ISO/IEC 42001 provide the organizational structure for this policy layer, while OWASP LLM Top 10 provides a security taxonomy for risks.  Policies must be translated into guardrail rules that are actionable: detect prompt injection in untrusted data, validate output schemas before tools execute, and block unsafe categories. This translation is non-trivial because LLM outputs are probabilistic and context-sensitive. A policy such as “never leak secrets” must be expressed as a chain of checks: input scanning for malicious prompts, context segmentation for untrusted data, output moderation to catch leakage, and tool gating to prevent exfiltration. Each check introduces uncertainty and cost, which means policy must include thresholds, confidence levels, and escalation paths.

Detectors such as Prompt Guard, Lakera Guard, or Rebuff provide risk scores and categories for injection attempts.  These detectors are probabilistic and therefore require calibration. The control plane must normalize detector outputs into a shared risk scale, define what “block” vs “review” means, and log the decision context for later auditing. Output guardrails such as Llama Guard or OpenAI moderation detect unsafe content in generated responses.  These checks must be aligned to your own taxonomy; the model’s categories may not map exactly to your policy. This is why evaluation and red-teaming are crucial: without testing, you do not know if your thresholds or taxonomy mappings are effective.

Structured output validation adds determinism to a probabilistic system. Guardrails AI uses validators and schema checks to ensure outputs conform to an expected structure, enabling safer tool calls and data extraction.  NeMo Guardrails extends this by introducing Colang, a flow language that constrains dialogue paths and allows explicit safety steps, such as mandatory disclaimers or confirmation prompts.  These frameworks provide building blocks, but they do not decide how to integrate them into a business context. For example, a schema validator can ensure a tool call is syntactically correct, but only policy can decide if that tool call should be allowed at all. This is why tool permissioning and sandboxing are critical complement pieces that most guardrails frameworks do not provide natively.

Evaluation is the evidence layer. Tools like garak and OpenAI Evals allow you to run red-team tests and custom evaluation suites to measure whether guardrails are actually working.  Without these tests, guardrails may create a false sense of security. Monitoring and telemetry are the final layer: you must log guardrail decisions, measure false positives and negatives, and track drift over time. Guardrails AI supports observability integration via OpenTelemetry, which can feed monitoring dashboards for guardrail KPIs.  The control plane is therefore a loop: policies drive controls, controls generate evidence, and evidence updates policies. This loop is the only sustainable way to manage guardrails in production.

How this fit on projects

You will apply this control-plane model directly in §5.4 and §5.11 and validate it in §6.

Definitions & key terms

Control plane: The policy-driven layer that decides what an agent may do.
Detector: A model or rule that assigns risk categories or scores. 
Validator: A structured check that enforces schema or constraints. 
Tool gating: Permissions and constraints for tool execution.
Evaluation suite: A set of tests that measure guardrails effectiveness. 

Mental model diagram

Policy -> Detectors -> Validators -> Tool Gate -> Output
 ^ | | | |
 | v v v v
Evidence <- Logs <- Thresholds <- Decisions <- Monitoring

How it works (step-by-step)

Define policy risks and acceptable thresholds.
Select detectors and validators aligned to those risks.
Normalize detector outputs and enforce schema rules.
Apply tool permissions based on risk and context.
Log decisions and run evaluation suites continuously.

Minimal concrete example

Guardrail Decision Record
- input_source: retrieved_doc
- detector: prompt_injection
- score: 0.84
- policy_action: block
- tool_gate: deny
- audit_id: 2026-01-03-0001

Common misconceptions

“A single framework solves guardrails end-to-end.”
“Moderation is enough to prevent prompt injection.”
“Validation guarantees correctness without policy.”

Check-your-understanding questions

Why is a policy layer required in addition to detectors?
How do validators reduce tool misuse risk?
Why is evaluation necessary even if detectors exist?

Check-your-understanding answers

Detectors provide signals, but policy decides actions and thresholds.
Validators ensure structured, safe tool inputs before execution.
Detectors can fail or drift; evaluation reveals blind spots.

Real-world applications

Enterprise assistants with access to sensitive data
RAG systems ingesting third-party documents
Autonomous workflows with high-impact tools

Where you’ll apply it

See §5.4 and §6 in this file.
Also used in: P02-prompt-injection-firewall.md, P03-content-safety-gate.md, P08-policy-router-orchestrator.md.

References

NIST AI RMF 1.0. 
ISO/IEC 42001:2023 AI Management Systems. 
OWASP LLM Top 10 v1.1. 
Guardrails AI framework. 
NeMo Guardrails and Colang. 
Prompt Guard model card. 
Llama Guard documentation. 
garak LLM scanner. 
OpenAI Evals. 

Key insights Guardrails are a control plane, not a single model or API.

Summary A layered control plane combines policy, detection, validation, and evaluation into a continuous safety loop.

Homework/Exercises to practice the concept

Draft a policy map with three risks and a detector for each.
Define a monitoring dashboard with three guardrail KPIs.

Solutions to the homework/exercises

Example risks: injection, data leakage, tool misuse; KPIs: block rate, false positives, tool denial rate.

3. Project Specification

3.1 What You Will Build

A RAG sanitization layer that scans retrieved docs before they enter the prompt.

3.2 Functional Requirements

Ingest retrieved documents
Run injection detection on doc text 
Apply provenance trust rules
Quarantine or allow docs

3.3 Non-Functional Requirements

Performance: Sanitization under 2 seconds per document
Reliability: Deterministic decisions for identical docs
Usability: Clear provenance reporting

3.4 Example Usage / Output

$ rag-guard scan --doc invoice.pdf
Decision: BLOCK
Reason: injection_detected

3.5 Data Formats / Schemas / Protocols

Provenance record: {doc_id, source, trust_score, action}

3.6 Edge Cases

Documents with mixed trusted/untrusted sections
Very large documents
Documents in multiple languages

3.7 Real World Outcome

This section is the golden reference. You will compare your output against it.

3.7.1 How to Run (Copy/Paste)

Provide document and source metadata
Run sanitization CLI
Review quarantine list

3.7.2 Golden Path Demo (Deterministic)

Use a fixed doc set and expected allow/block decisions.

3.7.3 If CLI: Exact Terminal Transcript (Success)

$ rag-guard scan --doc doc_safe.txt
Action: ALLOW
Trust: high

3.7.4 Failure Demo (Deterministic)

$ rag-guard scan --doc doc_malicious.txt
Action: BLOCK
Reason: injection_detected

Exit codes: 0 on allow, 2 on block

4. Solution Architecture

A RAG sanitizer sits between retrieval and prompt assembly, enforcing trust rules.

4.1 High-Level Design

┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Input │────▶│ Policy │────▶│ Output │
│ Handler │ │ Engine │ │ Reporter │
└─────────────┘ └─────────────┘ └─────────────┘

4.2 Key Components

Component	Responsibility	Key Decisions
Retrieval Input	Provides doc text	Must include source metadata
Injection Detector	Scans doc text	Source-aware threshold
Provenance Engine	Computes trust score	Allowlist/denylist

4.4 Data Structures (No Full Code)

Doc record fields: doc_id, source, text_hash, detection_score, action

4.4 Algorithm Overview

Load document
Detect injection
Compute trust score
Allow/block

5. Implementation Guide

5.1 Development Environment Setup

mkdir rag-guard && cd rag-guard

5.2 Project Structure

project/
├── ingest/
├── detectors/
├── provenance/
└── logs/

5.3 The Core Question You’re Answering

“How do I prevent retrieved documents from overriding my agent’s policy?”

Sanitization protects against indirect injection and untrusted sources.

5.4 Concepts You Must Understand First

Indirect prompt injection 
Provenance trust models

5.5 Questions to Guide Your Design

Trust rules
- Which sources are allowed?
- What is the quarantine policy?
Redaction
- Strip instructions or block?

5.6 Thinking Exercise

Document Trust Matrix

Assign trust scores to three data sources.

Questions to answer:

Which is highest risk?
Which can be allowed with logging?

5.7 The Interview Questions They’ll Ask

“What is indirect prompt injection?”
“How do you design provenance policies?”
“Why scan trusted sources?”
“How do you handle large documents?”
“How do you test RAG safety?”

5.8 Hints in Layers

Hint 1: Start with allowlists Only allow known sources.

Hint 2: Scan all docs Even trusted docs can be compromised.

Hint 3: Add quarantine Block and log suspicious docs.

Hint 4: Log metadata Capture source and timestamp.

5.9 Books That Will Help

Topic	Book	Chapter
Injection detection	Prompt Guard model card	Injection label 
Risk taxonomy	OWASP LLM Top 10	Prompt Injection 

5.10 Implementation Phases

Phase 1: Retrieval Ingestion (2-3 days)

Goals: load docs with metadata. Tasks: ingest pipeline. Checkpoint: doc list ready.

Phase 2: Detection Layer (3-4 days)

Goals: scan doc text. Tasks: detector integration. Checkpoint: detection scores output.

Phase 3: Provenance Policy (3-4 days)

Goals: trust scoring. Tasks: allowlist/denylist. Checkpoint: allow/block decisions.

5.11 Key Implementation Decisions

Decision	Options	Recommendation	Rationale
Block vs redact	block or redact	block high-risk	safest
Trust scoring	numeric vs categorical	categorical	simpler

6. Testing Strategy

6.1 Test Categories

Category	Purpose	Examples
Unit Tests	Validate core logic	Input classification, schema checks
Integration Tests	Validate end-to-end flow	Full guardrail pipeline
Edge Case Tests	Validate unusual inputs	Long prompts, empty outputs

6.2 Critical Test Cases

Trusted doc: allow
Injected doc: block
Unknown source: quarantine

6.3 Test Data

Doc set: 5 trusted, 5 untrusted, 5 injected

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall	Symptom	Solution
Trusting source blindly	Injection passes	Always scan
Over-blocking	Useful docs blocked	Adjust trust policy

7.2 Debugging Strategies

Inspect decision logs to see which rule triggered a block.
Replay deterministic test cases to reproduce failures.

7.3 Performance Traps

Avoid scanning the same doc repeatedly; cache results.

8. Extensions & Challenges

8.1 Beginner Extensions

Add redaction mode
Support more file types

8.2 Intermediate Extensions

Add source reputation scores
Integrate with policy router

8.3 Advanced Extensions

Real-time provenance dashboard
Multi-tenant trust policies

9. Real-World Connections

9.1 Industry Applications

Enterprise RAG: Blocks malicious documents
Knowledge assistants: Ensures trustworthy sources

Prompt Guard: Injection detection model 
Lakera Guard: Injection detection API 

9.3 Interview Relevance

RAG security: Trust and provenance design
Prompt injection: Indirect injection mitigation

10. Resources

10.1 Essential Reading

Prompt Guard model card. 
OWASP LLM Top 10. 

10.2 Video Resources

RAG security talks
Prompt injection demos

10.3 Tools & Documentation

Prompt Guard documentation. 
OWASP LLM Top 10. 

Project 2: Prompt Injection Firewall
Project 8: Policy Router Orchestrator

11. Self-Assessment Checklist

11.1 Understanding

I can explain the control plane layers without notes
I can justify every policy threshold used
I understand the main failure modes of this guardrail

11.2 Implementation

All functional requirements are met
All critical test cases pass
Edge cases are handled

11.3 Growth

I documented lessons learned
I can explain this project in an interview

12. Submission / Completion Criteria

Minimum Viable Completion:

Sanitization implemented
Provenance policy defined
Decisions logged

Full Completion:

All minimum criteria plus:
Quarantine workflow added
False positive rate measured

Excellence (Going Above & Beyond):

Source reputation model
Automated provenance dashboard