← Back to all projects

POSTMORTEM QUALITY LEARNING CULTURE MASTERY

In the complex, interconnected systems that define modern technology, failure is not an option; it's an inevitability. The true measure of an organization's resilience isn't its ability to prevent all incidents, but its capacity to learn from them. Historically, industries like aviation and healthcare pioneered rigorous incident analysis to prevent catastrophic failures. In the software world, this evolved into the practice of postmortems.

Learn Postmortem Quality & Learning Culture: From Zero to Learning Master

Goal: Deeply understand the principles and practices of high-quality postmortems, moving beyond mere incident reports to cultivate a robust learning culture within organizations. You will learn to facilitate blameless reviews, identify systemic weaknesses, and implement lasting improvements that enhance reliability, foster psychological safety, and drive continuous organizational learning.

Why Postmortem Quality & Learning Culture Matters

In the complex, interconnected systems that define modern technology, failure is not an option; it’s an inevitability. The true measure of an organization’s resilience isn’t its ability to prevent all incidents, but its capacity to learn from them. Historically, industries like aviation and healthcare pioneered rigorous incident analysis to prevent catastrophic failures. In the software world, this evolved into the practice of postmortems.

However, not all postmortems are created equal. A “blame culture” postmortem focuses on finding a scapegoat, leading to fear, concealment of information, and a cycle of repeated failures. In contrast, a “learning culture” postmortem, rooted in blameless principles, transforms incidents into invaluable opportunities for growth. It shifts the focus from “who” to “what” and “why,” uncovering the systemic factors that contribute to incidents and empowering teams to implement meaningful, preventative changes.

This knowledge unlocks:

  • Higher Availability: Systemic fixes prevent “butterfly effect” failures from recurring.
  • Improved Morale: Developers stop fearing the “3 AM pager” because they know failure leads to improvement, not punishment.
  • Faster Innovation: Teams with high psychological safety take more calculated risks.
  • Organizational Intelligence: Knowledge trapped in individual heads becomes shared company assets.

Core Concept Analysis

The Incident Lifecycle: A Continuous Loop of Improvement

Incidents are not isolated events with a clear beginning and end. They are part of a continuous lifecycle that, when managed effectively, drives organizational learning and resilience. A high-quality postmortem is the critical “learning” phase of this cycle, feeding insights back into preparation and prevention.

          +-------------------------------------------------+
          |                                                 |
          |    +-----------------+     +-----------------+  |
          |    |   Preparation   | --> |    Detection    |  |
          |    | (Tools, Training)|     | (Monitoring)    |  |
          |    +-----------------+     +-----------------+  |
          |            ^                        |           |
          |            |                        v           |
          |    +-----------------+     +-----------------+  |
          |    |    Learning     |     |   Containment   |  |
          |    | (Systemic Fixes)|     | (Stop the Bleed)|  |
          |    +-----------------+     +-----------------+  |
          |            ^                        |           |
          |            |                        v           |
          |    +-----------------+     +-----------------+  |
          |    |   Postmortem    | <-- |    Recovery     |  |
          |    | (Blameless Rev) |     | (Restore Serv)  |  |
          |    +-----------------+     +-----------------+  |
          |                                                 |
          +-------------------------------------------------+

Blameless Culture: The “Second Story”

A blameless culture acknowledges that complex systems fail in complex ways, and human error is often a symptom, not the root cause. It seeks the “Second Story”—the context, the conflicting priorities, and the missing information that made the person’s action make sense at the time.

Blame Culture (The First Story):         Blameless Culture (The Second Story):
"Bob deleted the database."             "The database UI allowed a one-click delete
                                         without a confirmation dialog, and the
                                         backups were failing silently for months."

Focus: Person                           Focus: System & Environment
Outcome: Punishment/Fear                Outcome: Safety guards & better monitoring

Systemic Thinking: The Swiss Cheese Model

Systemic thinking means looking beyond the immediate, obvious cause (the “proximate cause”) to uncover the underlying conditions and interactions. James Reason’s “Swiss Cheese Model” illustrates how incidents occur when holes in multiple layers of defense align.

      Hazard ---> [ S1 ] ---> [ S2 ] ---> [ S3 ] ---> FAILURE
                    |           |           |
                    O           |           O   <-- Holes (Latent Weaknesses)
                    |           O           |
                    |           |           |

S1: Monitoring      S2: Code Review     S3: Deployment Process
  • Latent Conditions: Weaknesses hidden in the system (e.g., outdated docs, technical debt).
  • Active Failures: The immediate triggers (e.g., a wrong command).
  • Goal: Add layers or shrink the holes.

Psychological Safety: The Foundation

Psychological safety is the shared belief that a team is safe for interpersonal risk-taking. Without it, people hide mistakes, and you lose the data necessary to fix the system.

High Psych Safety:                      Low Psych Safety:
"I missed this check."                  "I hope nobody notices I missed this."
       |                                       |
       v                                       v
System is fixed.                        System remains broken.
                                        Failure repeats.

Concept Summary Table

Concept Cluster What You Need to Internalize
Blamelessness Human error is a starting point for investigation, not a conclusion. Seek “why it made sense” to the person.
Systemic Investigation Use the Swiss Cheese Model to find latent weaknesses in tools, processes, and environment.
Psychological Safety The prerequisite for honest disclosure. Teams must feel safe to admit mistakes without reprisal.
Actionable Learning A postmortem is a failure if it doesn’t produce concrete, trackable improvements (systemic fixes).
Knowledge Sharing Incident learnings must be broadcast across the organization to prevent similar failures in other teams.

Deep Dive Reading by Concept

This section maps each concept to specific book chapters. Read these alongside the projects.

The Theory of Failure & Systems

Concept Book & Chapter
Resilience Engineering “The Field Guide to Human Error Investigations” by Sidney Dekker — Ch. 1: “The Old View and the New View”
The Second Story “Site Reliability Engineering” by Betsy Beyer — Ch. 15: “Postmortem Culture: Learning from Failure”
Systems Thinking “The Fifth Discipline” by Peter Senge — Ch. 4: “The Laws of the Fifth Discipline”

Cultural Foundations

Concept Book & Chapter
Psychological Safety “The Fearless Organization” by Amy Edmondson — Ch. 1: “The Foundation”
The Five Ideals “The Unicorn Project” by Gene Kim — Ch. 13: “Blameless Post-Mortems”
Reliability Culture “Accelerate” by Nicole Forsgren — Ch. 3: “Measuring and Changing Culture”

Essential Reading Order

  1. The Mindset Shift (Day 1):
    • SRE Book Ch. 15 (Google’s approach)
    • Field Guide Ch. 1 ( Dekker’s “Old View vs New View”)
  2. The Tools of Analysis (Week 1):
    • The Fearless Organization Ch. 1 & 2
    • The Unicorn Project Ch. 13

Project 1: The Blame-Free Template Engine

  • File: POSTMORTEM_TEMPLATE_GEN.py
  • Main Programming Language: Python
  • Alternative Programming Languages: JavaScript, Ruby
  • Coolness Level: Level 2: Practical but Forgettable
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 1: Beginner
  • Knowledge Area: Document Engineering / UX Design
  • Software or Tool: Markdown, Jinja2
  • Main Book: “Site Reliability Engineering” (Google)

What you’ll build: A CLI tool that generates a structured Markdown postmortem template based on incident severity and type, specifically designed to steer the author away from blame.

Why it teaches postmortem quality: By codifying the structure (e.g., replacing a “Who” section with “Latent Systemic Factors”), you learn how architecture influences culture. You’ll discover that a well-designed form can prevent “lazy” investigations.

Core challenges you’ll face:

  • Information Architecture → Deciding which fields are mandatory for a “quality” report.
  • Steering Behavior → Crafting prompts that ask “What made this action make sense?” rather than “What went wrong?”.
  • Flexibility → Supporting different incident types (e.g., security vs. availability).

Key Concepts

  • SRE Postmortem Checklist: [Google SRE Book Ch. 15]
  • The “Five Whys” (and its pitfalls): [The Field Guide to Human Error - Sidney Dekker]

Difficulty: Beginner Time estimate: Weekend Prerequisites: Basic Python strings/file I/O.

Real World Outcome

You will have a standardized tool that teams use to start their investigations. It ensures no postmortem starts with a blank page and that every report includes the “Second Story.”

Example Output:

$ python gen_pm.py --type availability --severity SEV1
[INFO] Generating Postmortem Template for SEV1 Availability Incident...
[INFO] Created: 2024-12-28_database_outage_draft.md

$ cat 2024-12-28_database_outage_draft.md
## Executive Summary
...
## The Second Story: Context of the Decision
> Use this section to explain why the actions taken made sense at the time.
> Avoid: "The operator forgot..."
> Use: "The operator's dashboard lacked the X metric which would have..."
...
## Latent Conditions (The Swiss Cheese Holes)
1. [ ]

The Core Question You’re Answering

“How can we design our tools to make blamelessness the path of least resistance?”

Before you write any code, sit with this question. Most bad postmortems happen because people follow the path of least resistance, which is usually “human error.”

Concepts You Must Understand First

Stop and research these before coding:

  1. The First Story vs. The Second Story
    • What is the “First Story” of an accident?
    • Why is the “Second Story” harder to find but more valuable?
    • Book Reference: “The Field Guide to Human Error Investigations” Ch. 1 - Sidney Dekker
  2. Proximate vs. Root vs. Systemic Causes
    • Why do many SREs hate the term “Root Cause”?
    • What is a “Latent Condition”?
    • Book Reference: “Site Reliability Engineering” Ch. 15 - Google

Questions to Guide Your Design

  1. Facilitation through Structure
    • What section comes after “Timeline”? If it’s “Fixes,” are you skipping the “Analysis”?
    • How do you prompt the user to look at tools instead of people?
  2. Severity-Based Depth
    • Does a SEV3 (minor) need the same depth as a SEV1 (catastrophic)?
    • How do you balance the “Work to Learn” ratio?

Thinking Exercise

The Prompt Rewrite

Before coding, look at these standard “Blame” prompts and rewrite them to be “Systemic”:

  1. “List the person who initiated the deployment.”
  2. “Why did the developer bypass the test suite?”
  3. “What mistake was made during the configuration change?”

Questions while rewriting:

  • Does your rewrite focus on the environment or the actor?
  • Does your rewrite encourage finding a fix or a culprit?

The Interview Questions They’ll Ask

  1. “How do you ensure postmortems don’t turn into finger-pointing exercises?”
  2. “Why is a ‘Root Cause’ often a misleading concept in complex systems?”
  3. “What are the most important sections of a postmortem report and why?”
  4. “How do you handle a situation where a manager insists on ‘accountability’ (punishment) for an incident?”
  5. “What is the difference between an ‘Action Item’ and a ‘Systemic Fix’?”

Hints in Layers

Hint 1: Start with the SRE Template Look at the open-source Google or Etsy postmortem templates. Use these as your baseline.

Hint 2: Focus on the Prompts The value isn’t in the Markdown structure, but in the comments inside the template that guide the author.

Hint 3: Use a Templating Engine Don’t just use f-strings. Use Jinja2 or similar to handle logic like if severity == 'SEV1': include_deep_analysis_section().

Hint 4: Validation Can you add a small “Quality Check” script that greps the finished report for banned words like “careless” or “negligent”?

Books That Will Help

Topic Book Chapter
Template Structure “Site Reliability Engineering” by Google Ch. 15
Steering Language “The Field Guide to Human Error” by Sidney Dekker Ch. 1

Project 2: Incident Timeline Scraper

  • File: TIMELINE_EXTRACTOR.py
  • Main Programming Language: Python
  • Alternative Programming Languages: Go, Node.js
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 2. The “Micro-SaaS / Pro Tool”
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Data Mining / APIs
  • Software or Tool: Slack API, Discord API, or Loggly API
  • Main Book: “How Linux Works” (for timestamp/log understanding)

What you’ll build: A tool that extracts messages and events from a specific time window in Slack/Discord incident channels to build a factual, objective timeline.

Why it teaches postmortem quality: Human memory is unreliable and prone to “hindsight bias” (believing we knew more at the time than we did). An objective timeline is the anchor for a blameless review.

Core challenges you’ll face:

  • Timezone Normalization → Dealing with UTC vs. local time in logs vs. chat.
  • Noise Filtering → Distinguishing “We are looking at X” from “Server is down.”
  • Context Preservation → Pulling threads or replies that contain critical decision logic.

Key Concepts

  • Hindsight Bias: [The Field Guide to Human Error - Sidney Dekker]
  • API Rate Limiting: [Standard Engineering Practice]

Difficulty: Intermediate Time estimate: 1 week Prerequisites: API authentication knowledge (OAuth/Tokens), JSON parsing.

Real World Outcome

An objective, timestamped CSV or Markdown table that serves as the “source of truth” for the postmortem meeting.

Example Output:

| Timestamp (UTC) | Source | Event/Message |
|-----------------|--------|---------------|
| 14:02:11        | PagerDuty | Alert: DB Latency High |
| 14:03:45        | Alice  | "Looking at the query logs now." |
| 14:05:20        | Grafana | CPU Spike on Web-01 |
| 14:10:00        | Bob    | "I'm going to restart the service." |

Project 3: The “Blame-Scanner” Linter

  • File: BLAME_LINTER.py
  • Main Programming Language: Python (with Spacy or NLTK)
  • Alternative Programming Languages: Rust, JavaScript
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 2. The “Micro-SaaS / Pro Tool”
  • Difficulty: Level 3: Advanced
  • Knowledge Area: NLP / Static Analysis
  • Software or Tool: Spacy, Regex
  • Main Book: “The Field Guide to Human Error Investigations”

What you’ll build: A linter for postmortem drafts that identifies “Blame Language” and “Hindsight Bias” patterns, suggesting more constructive, systemic ways to phrase findings.

Why it teaches postmortem quality: It forces you to codify the linguistic differences between blame and learning. You’ll have to define exactly what a “blaming sentence” looks like.

Core challenges you’ll face:

  • Linguistic Nuance → Distinguishing between “The user did X” (fact) and “The user failed to do X” (judgment).
  • Suggestion Engine → Not just flagging errors, but providing “Systemic Alternatives.”
  • Hindsight Detection → Identifying phrases like “should have known” or “it was obvious that.”

Key Concepts

  • Counterfactuals: [The Field Guide to Human Error - Sidney Dekker]
  • Language and Safety Culture: [Amy Edmondson - The Fearless Organization]

Difficulty: Advanced Time estimate: 2 weeks Prerequisites: Basic NLP (tokenization, POS tagging), Regex mastery.

Real World Outcome

A CI/CD check or local CLI tool that “grades” a postmortem draft based on its adherence to blameless principles.

Example Output:

$ blame-scan report_draft.md

[WARNING] Line 45: "The engineer should have checked the logs before restarting."
  -> Category: Hindsight Bias / Counterfactual
  -> Advice: Focus on what information was actually available to the engineer at 14:10.
  -> Suggested Phrasing: "The dashboard used by the engineer did not display the log-tailing view, which contained the error signature."

[CRITICAL] Line 12: "This was caused by operator negligence."

  -> Category: Blame / Judgmental Language

  -> Advice: Remove judgmental adjectives. Focus on system design.

Project 4: Systemic Action-Item Tracker

  • File: ACTION_ITEM_TRACKER.md

  • Main Programming Language: Node.js / React

  • Alternative Programming Languages: Python (Django/Flask), Go

  • Coolness Level: Level 2: Practical but Forgettable

  • Business Potential: 3. The “Service & Support” Model

  • Difficulty: Level 2: Intermediate

  • Knowledge Area: Project Management / Databases

  • Software or Tool: SQLite/PostgreSQL, Jira/GitHub API

  • Main Book: “The Unicorn Project” (Gene Kim)

What you’ll build: A specialized dashboard that tracks postmortem action items, categorizing them by “Type of Fix” (e.g., Tooling, Process, Monitoring) and linking them to Error Budgets.

Why it teaches postmortem quality: Many postmortems are “write-only”—they are created and forgotten. This project focuses on the outcome. You’ll learn how to prioritize “Preventative” fixes over “Quick Patches.”

Core challenges you’ll face:

  • Data Modeling → Relating incidents to multiple fixes and fixing teams.

  • SLA/SLO Integration → Automatically flagging if a postmortem’s SEV1 action items are overdue based on company policy.

  • Categorization Logic → Differentiating between a “Temporary Mitigation” and a “Systemic Fix.”

Key Concepts

  • Error Budgets: [Google SRE Book Ch. 3]

  • The “Work to Learn” Ratio: [Adaptive Capacity Labs - John Allspaw]

Difficulty: Intermediate

Time estimate: 2 weeks

Prerequisites: Basic web dev (frontend + backend), SQL.

Real World Outcome

A live dashboard showing which teams are successfully closing their “Learning Debt” and which systemic categories are most frequently targeted.

Example Output:


# Web Dashboard View:

Incident: "Payment Gateway Timeout"

Status: RESOLVED

Systemic Fixes:

  [ID 102] Add circuit breaker to Payment Client (TOOLING) - COMPLETED

  [ID 103] Automate load-test for payment flow (PROCESS) - IN PROGRESS

  [ID 104] Update PagerDuty rotation to include Lead (CULTURE) - PLANNED



Metrics:

  90% of SEV1 items closed within 30 days.

  Focus Area: 60% of fixes are 'Monitoring', only 10% are 'Architecture'.

Project 5: Postmortem Metrics Dashboard

  • File: POSTMORTEM_METRICS.py

  • Main Programming Language: Python

  • Alternative Programming Languages: Go, SQL

  • Coolness Level: Level 3: Genuinely Clever

  • Business Potential: 3. The “Service & Support” Model

  • Difficulty: Level 2: Intermediate

  • Knowledge Area: Data Visualization / SRE

  • Software or Tool: Grafana, Prometheus (or just CSV + Matplotlib)

  • Main Book: “Accelerate” (Nicole Forsgren)

What you’ll build: A visualization suite that tracks the health of the postmortem process itself, not just the incidents.

Why it teaches postmortem quality: You’ll learn to measure culture. If “Time to Postmortem Published” is high, your learning culture is stalling. If “Repeat Incidents” are high, your postmortem quality is low.

Core challenges you’ll face:

  • Defining Quality Metrics → How do you measure a “good” postmortem automatically? (e.g., word count, number of action items, cross-team attendance).

  • Data Aggregation → Pulling data from Jira, GitHub, and Postmortem Markdown files.

  • Trend Analysis → Detecting if the organization is getting better or worse at learning over time.

Key Concepts

  • Westrum Organizational Culture: [Accelerate Ch. 3]

  • Learning from Incidents (LFI) Metrics: [Jeli.io / Nora Jones]

Difficulty: Intermediate

Time estimate: 1-2 weeks

Prerequisites: Data visualization basics, simple statistics.

Real World Outcome

A “State of Learning” report generated monthly that highlights which teams are the best at distilling and applying lessons.

Project 6: The Mock Incident Simulator

  • File: INCIDENT_SIMULATOR.sh

  • Main Programming Language: Bash / Python

  • Alternative Programming Languages: Go

  • Coolness Level: Level 4: Hardcore Tech Flex

  • Business Potential: 1. The “Resume Gold”

  • Difficulty: Level 3: Advanced

  • Knowledge Area: Chaos Engineering / Education

  • Software or Tool: Docker, Chaos Mesh

  • Main Book: “Operating Systems: Three Easy Pieces” (for system failure modes)

What you’ll build: A “Chaos” script that breaks a local Docker-compose environment in a specific, subtle way, followed by a guided “Training Postmortem” session script for a team.

Why it teaches postmortem quality: Building the failure yourself teaches you how latent conditions work. Facilitating the session teaches you how to handle the social dynamics of blamelessness.

Core challenges you’ll face:

  • Repeatable Failure → Making a failure that isn’t too obvious but is discoverable.

  • Guided Inquiry → Creating a script for a “Facilitator” that includes Socratic questions.

  • Environment Isolation → Ensuring the “Incident” doesn’t escape the lab environment.

Key Concepts

  • Chaos Engineering: [Principlesofchaos.org]

  • Socratic Facilitation: [The Fearless Organization]

Difficulty: Advanced

Time estimate: 2 weeks

Prerequisites: Docker, Linux systems knowledge, basic shell scripting.

Real World Outcome

A “Postmortem Workshop Kit” you can run for your team. You trigger a “database connection leak” and then lead them through the process of finding it and writing a blameless report.

Example Simulator Output:


$ ./sim_incident.sh start --scenario "slow_leak"

[OK] Environment up.

[OK] Injecting Latent Condition: Max connections set to 5.

[OK] Starting Traffic Generator...

[ALERT] 14:02:00 - 500 Errors detected on Web-API!

[FACILITATOR] Task: Open your Postmortem Template and start the Timeline.

Project 7: Knowledge Sharing “Digest” Generator

  • File: LEARNING_DIGEST.py

  • Main Programming Language: Python

  • Alternative Programming Languages: Node.js, Go

  • Coolness Level: Level 3: Genuinely Clever

  • Business Potential: 2. The “Micro-SaaS / Pro Tool”

  • Difficulty: Level 2: Intermediate

  • Knowledge Area: Knowledge Management / Web Dev

  • Software or Tool: Markdown, Static Site Generator (Hugo/Eleventy)

  • Main Book: “The Pragmatic Programmer”

What you’ll build: A tool that crawls a directory of postmortem Markdown files, extracts “Key Lessons” and “Systemic Fixes,” and generates a searchable, internal “Learning Site.”

Why it teaches postmortem quality: It emphasizes the “Learning” in “Postmortem.” If a postmortem is written but never read by anyone outside the team, its value is halved. This forces you to think about audience and summarization.

Core challenges you’ll face:

  • Structured Data Extraction → Using regex or front-matter to pull metadata from unstructured Markdown.

  • Searchability → Implementing a simple client-side search (e.g., Fuse.js) for finding incidents by “Tags” (e.g., #network, #database).

  • Incentive Design → Creating a “Summary” field in the template that is compelling enough for people to actually click and read.

Key Concepts

  • Organizational Memory: [The Fifth Discipline - Peter Senge]

  • Static Site Generation: [Modern Web Patterns]

Real World Outcome

A polished internal portal where developers can search for “DNS” and see all past incidents, their fixes, and avoid making the same mistakes in their own projects.

Example Output:


$ ./generate_digest.sh

[INFO] Scanning /docs/postmortems...

[INFO] Found 15 reports.

[INFO] Generating site at /site/index.html...



# Site View:

# "Top Learnings this Month"

# 1. We found that our Go library doesn't handle timeouts by default (See SEV2: Gateway Timeout)

# 2. Redis cluster failovers take 30s, not 5s as documented.

The Core Question You’re Answering

“How do we turn a team’s failure into a company’s asset?”

Before you write any code, sit with this question. Most institutional knowledge is lost when people leave. This project aims to make that knowledge permanent and searchable.

Concepts You Must Understand First

  1. Metadata vs. Content

    • How do you tag a postmortem so it’s useful to others?

    • Book Reference: “The GNU Make Book” (for understanding file processing) - John Graham-Cumming

  2. Information Scent

    • What makes a headline “clickable” for an engineer who is busy?

    • Resource: “Designing Data-Intensive Applications” Ch. 1 (Reliability context)

Questions to Guide Your Design

  1. Accessibility

    • How can you make the digest easy to consume in 5 minutes?

    • Should it be a website, or a PDF emailed to everyone?

  2. Structure

    • What’s more important: the timeline or the “Action Items”?

Thinking Exercise

The Learning Extraction

Look at a sample postmortem report. Try to write a 3-sentence summary that would make another engineer say, “I need to check my code for this.”

Questions while summarizing:

  • Did you include the specific failure mode?

  • Did you mention the systemic fix?

The Interview Questions They’ll Ask

  1. “How do you ensure that incident learnings are shared across the whole engineering org?”

  2. “What are the trade-offs between a detailed postmortem and a high-level summary?”

  3. “How do you manage the privacy of individuals while still sharing technical failures?”

Hints in Layers

Hint 1: Use Front-matter

Add a YAML block at the top of your postmortems for title, severity, tags, and summary.

Hint 2: Markdown Parsing

Use a library like mistune or markdown-it to parse the files into structured objects.

Hint 3: Search

Don’t build a full database. Use a static index file and a JS library to provide instant search.

Project 8: The “Counterfactual” Analyzer

  • File: COUNTERFACTUAL_GUIDE.md

  • Main Programming Language: None (Design/Process Project)

  • Alternative Programming Languages: Web App (React)

  • Coolness Level: Level 4: Hardcore Tech Flex

  • Business Potential: 1. The “Resume Gold”

  • Difficulty: Level 3: Advanced (Conceptually)

  • Knowledge Area: Cognitive Science / Investigation

  • Software or Tool: Miro, Whimsical, or a Custom Web Form

  • Main Book: “The Field Guide to Human Error Investigations” (Dekker)

What you’ll build: A structured, interactive tool or decision tree that guides investigators through “Counterfactual Analysis” without falling into “Hindsight Bias.”

Why it teaches postmortem quality: You’ll learn to ask “What could have happened?” in a way that reveals system weaknesses rather than individual failings. It forces you to map out “The path not taken.”

Core challenges you’ll face:

  • Hindsight Trap → Preventing users from saying “They should have just…”

  • Alternative Path Mapping → Visualizing the decision points during an incident.

  • Data Capture → Recording the reasons why a better path wasn’t taken (e.g., “The alarm didn’t sound”).

Real World Outcome

A “Decision Map” of an incident that shows where the system misled the humans, rather than where the humans failed the system.

Project 9: Safety Culture Assessment Bot

  • File: SAFETY_BOT.py

  • Main Programming Language: Python (Slack SDK)

  • Alternative Programming Languages: Node.js, Go

  • Coolness Level: Level 3: Genuinely Clever

  • Business Potential: 2. The “Micro-SaaS / Pro Tool”

  • Difficulty: Level 2: Intermediate

  • Knowledge Area: Social Engineering / Metrics

  • Software or Tool: Slack/Teams API, MongoDB

  • Main Book: “The Fearless Organization” (Amy Edmondson)

What you’ll build: A Slack bot that periodically sends anonymous 1-question polls to engineering teams to measure their “Psychological Safety Score” specifically regarding incident reporting.

Why it teaches postmortem quality: You can’t have a good postmortem without a safe culture. This project connects the “Soft Skills” of culture to “Hard Data.”

Core challenges you’ll face:

  • Anonymity Assurance → Ensuring users trust that their specific answers can’t be traced back to them.

  • Metric Selection → Choosing the right questions based on the Edmondson scale (e.g., “On this team, it is easy to speak up about problems”).

  • Visualization → Presenting trends over time to management without creating a “Blame Game” for low-scoring teams.

Real World Outcome

A “Culture Dashboard” that shows the correlation between psychological safety and incident recovery speed.

The Interview Questions They’ll Ask (Project 9)

  1. “How do you measure psychological safety in a quantitative way?”

  2. “What do you do if a team has a consistently low safety score?”

  3. “Why is anonymity critical for these types of metrics?”

  4. “How does safety culture directly impact system uptime?”

  5. “Can you have a ‘Blameless Postmortem’ in a ‘Low Safety’ culture?”

```

Project 10: The Postmortem “Black Box” Recorder

  • File: POSTMORTEM_BLACKBOX.py

  • Main Programming Language: Go

  • Alternative Programming Languages: Python, Rust

  • Coolness Level: Level 5: Pure Magic (Super Cool)

  • Business Potential: 5. The “Industry Disruptor”

  • Difficulty: Level 4: Expert

  • Knowledge Area: System Integration / Automation

  • Software or Tool: PagerDuty API, Prometheus, GitHub, Slack

  • Main Book: “Site Reliability Engineering” (Google)

What you’ll build: A system that listens for incident resolution events (e.g., from PagerDuty) and automatically assembles a “Black Box” zip file containing the chat history, relevant Grafana screenshots (via API), and a pre-filled Markdown postmortem draft with a generated timeline.

Why it teaches postmortem quality: It reduces the “Toil” of writing a postmortem. By automating the data collection, you free the engineers to focus on the deep analysis rather than the busy work.

Core challenges you’ll face:

  • Cross-Platform Correlation → Linking a PagerDuty ID to a specific Slack channel and GitHub PR.

  • Visual Capture → Using headless browsers (e.g., Playwright) to capture dashboard state at specific timestamps.

  • Workflow Orchestration → Managing long-running data collection tasks without losing state.

Real World Outcome

A “Postmortem Pack” that appears in a shared drive 5 minutes after an incident is closed, containing everything needed to start the review.

The Core Question You’re Answering

“How do we eliminate the ‘Toil’ of investigation so we can focus on the ‘Wisdom’ of analysis?”

Most engineers dread postmortems because they spend 4 hours copying and pasting timestamps. This project asks if automation can save the human soul of the investigation.

Concepts You Must Understand First

  1. API Interoperability

    • How do you map different ID types across PagerDuty, Slack, and GitHub?

    • Book Reference: “Enterprise Integration Patterns” by Gregor Hohpe

  2. State Management in Automation

    • What happens if the Slack scraper fails but the Grafana capture succeeds?

Questions to Guide Your Design

  1. Privacy & Security

    • What happens if the Slack channel contains private credentials that shouldn’t be in a permanent report?

    • How do you scrub sensitive data automatically?

  2. Data Retention

    • Where do these “Black Box” files live, and who has access?

Thinking Exercise

The Data Map

Draw a diagram showing every piece of data you want to collect and the API endpoint required to get it.

Questions:

  • Which piece of data is the hardest to get?

  • Which piece of data is the most likely to change if you wait 24 hours?

The Interview Questions They’ll Ask

  1. “How do you automate the collection of context without creating a mountain of noise?”

  2. “What are the biggest technical hurdles in cross-platform incident data aggregation?”

  3. “How can automation actually hurt the quality of a postmortem if not careful?”

Hints in Layers

Hint 1: Start with PagerDuty Webhooks

Use webhooks to trigger your script.

Hint 2: Headless Browsers

Use chromedp (for Go) or playwright to capture the Grafana dashboards at the exact time of the incident.

Hint 3: Slack Conversations History

Use the conversations.history API with oldest and latest timestamps matching the incident duration.

Project Comparison Table

Project Difficulty Time Depth of Understanding Fun Factor

|———|————|——|————————|————|

1. Template Engine Level 1 Weekend Medium Low
2. Timeline Scraper Level 2 1 Week Medium Medium
3. Blame Linter Level 3 2 Weeks High High
4. Action Item Tracker Level 2 2 Weeks Medium Low
5. Metrics Dashboard Level 2 1 Week Medium Medium
6. Mock Incident Sim Level 3 2 Weeks High High
7. Learning Digest Level 2 1 Week Medium Medium
9. Safety Bot Level 2 1 Week High Medium
10. Black Box Recorder Level 4 1 Month Very High Extreme

Recommendation

Start with Project 1 (The Template Engine). It is the easiest to implement but has the highest immediate impact on how you think about failure. Once you have a template, use Project 3 (The Blame Linter) to refine it. This combination will rewire your brain to stop looking for culprits and start looking for systems.

Final Overall Project: The “Learning Culture” Operating System

What you’ll build: A unified platform (The “Learning OS”) that integrates all the tools above. It should provide a single workflow for an engineer:

  1. Trigger: An incident is resolved.

  2. Collect: The “Black Box” automatically pulls logs, charts, and chats.

  3. Draft: The Linter helps the engineer write the “Second Story” in the Template Engine.

  4. Approve: A peer-review system for postmortems ensures quality.

  5. Close: Action items are synced to the company’s task tracker.

  6. Share: The Digest Generator broadcasts the results to the org.

This is the “Holy Grail” of Engineering Management. It turns failure from a source of stress into a streamlined, automated manufacturing process for organizational wisdom.

Summary

This learning path covers Postmortem Quality & Learning Culture through 10 hands-on projects. Here’s the complete list:

# Project Name Main Language Difficulty Time Estimate

|—|————–|—————|————|—————|

1 The Blame-Free Template Engine Python Beginner Weekend
2 Incident Timeline Scraper Python Intermediate 1 week
3 The “Blame-Scanner” Linter Python (NLP) Advanced 2 weeks
4 Systemic Action-Item Tracker Node/React Intermediate 2 weeks
5 Postmortem Metrics Dashboard Python Intermediate 1 week
6 The Mock Incident Simulator Bash/Python Advanced 2 weeks
7 Knowledge Sharing Digest Python Intermediate 1 week
8 The Counterfactual Analyzer Design Advanced 2 weeks
9 Safety Culture Assessment Bot Python Intermediate 1 week
10 The Postmortem Black Box Go Expert 1 month

For beginners: Start with projects #1, #2, and #5.

For intermediate: Jump to projects #3, #4, and #7.

For advanced: Focus on projects #6, #9, and #10.

Expected Outcomes

After completing these projects, you will:

  • Understand the deep linguistic and psychological differences between blame and learning.

  • Be able to facilitate high-stakes postmortem meetings for major outages.

  • Have a portfolio of tools that demonstrate your ability to scale SRE culture.

  • Know how to measure and improve the “Safety Culture” of any engineering team.

  • Move from being an engineer who “fixes bugs” to a leader who “improves systems.”

You’ll have built 10 working projects that demonstrate deep understanding of Learning Culture from first principles.