Project 7: Self-Critique and Repair Loop

Build a reflection loop that critiques outputs, identifies defects, and retries with targeted fixes.

Quick Reference

Attribute Value
Difficulty Level 3: Advanced
Time Estimate 12–20 hours
Language Python or JavaScript
Prerequisites Projects 2–6, eval basics
Key Topics reflection, retries, repair prompts, evals

Learning Objectives

By completing this project, you will:

  1. Implement critique pass with a rubric.
  2. Detect failures (missing fields, hallucinations).
  3. Generate targeted repair prompts.
  4. Limit retries with budgets.
  5. Measure improvement rates across runs.

The Core Question You’re Answering

“How can an agent catch and fix its own mistakes before users see them?”

The reflection loop is a reliability layer, not a replacement for validation.

Concepts You Must Understand First

Concept Why It Matters Where to Learn
Critique rubrics Objective error detection Evals guides
Retry budgets Prevent runaway costs Reliability patterns
Structured output Enables targeted fixes Schema validation docs
Self-consistency Reduces hallucinations LLM eval papers

Theoretical Foundation

Reflection as Error Correction

Generate -> Critique -> Repair -> Re-evaluate

The critique step must be independent, deterministic, and grounded in explicit criteria.

Project Specification

What You’ll Build

An agent that validates its output, critiques it against a rubric, and retries with fix instructions.

Functional Requirements

  1. Critique model with rubric
  2. Repair prompt generation
  3. Retry budget + termination
  4. Before/after comparison
  5. Improvement metrics

Non-Functional Requirements

  • Deterministic evaluation
  • Traceable repair logs
  • Safe fallback if retries fail

Real World Outcome

Example fix loop:

{
  "attempt": 1,
  "errors": ["missing citation", "unsupported claim"],
  "repair": "Add citations for claims 2 and 3"
}

After retry, the output is schema-valid and all claims are cited.

Architecture Overview

┌──────────────┐   output   ┌───────────────┐
│ Generator    │──────────▶│ Critique      │
└──────┬───────┘           └──────┬────────┘
       │ repair                         │
       ▼                                ▼
┌──────────────┐                 ┌──────────────┐
│ Repair Loop  │◀────────────────│ Validator    │
└──────────────┘                 └──────────────┘

Implementation Guide

Phase 1: Rubric + Critique (3–4h)

  • Define rubric criteria
  • Checkpoint: critique flags missing fields

Phase 2: Repair Loop (4–6h)

  • Generate fixes and retry
  • Checkpoint: output improves after retry

Phase 3: Metrics (4–6h)

  • Track success rate of repairs
  • Checkpoint: improvement report generated

Common Pitfalls & Debugging

Pitfall Symptom Fix
Over-critique outputs never pass relax rubric thresholds
Retry storms costs spike enforce max retries
Same error repeats no improvement inject explicit fixes

Interview Questions They’ll Ask

  1. Why use a separate critique pass instead of one prompt?
  2. How do you decide retry limits?
  3. How do you measure improvement objectively?

Hints in Layers

  • Hint 1: Start with a simple rubric of required fields.
  • Hint 2: Convert critique to explicit fix instructions.
  • Hint 3: Stop after success or max retries.
  • Hint 4: Compare before/after outputs to score improvement.

Learning Milestones

  1. Critique Works: errors detected reliably.
  2. Repairs Work: retries fix common issues.
  3. Measured: improvement rates reported.

Submission / Completion Criteria

Minimum Completion

  • Critique + single retry

Full Completion

  • Retry budgets + improvement metrics

Excellence

  • Multi-sample self-consistency
  • Automatic test generation

This guide was generated from project_based_ideas/AI_AGENTS_LLM_RAG/AI_AGENTS_PROJECTS.md.