← Back to all projects

DATA MESH ARCHITECTURE MASTERY

For decades, the standard for big data was the **Centralized Monolith**. Organizations funneled all data into a single Data Warehouse or Data Lake, managed by a specialized Data Team that became a permanent bottleneck. As the number of data sources exploded and business domains became more complex, this model broke.

Learn Data Mesh Architecture: From Monolith to Decentralized Ecosystem

Goal: Deeply understand the Data Mesh paradigm—shifting from centralized data lakes to a decentralized, domain-driven architecture. You will learn how to treat data as a product, implement federated governance through code, and build a self-serve platform that empowers domain teams while ensuring global interoperability and compliance.

Why Data Mesh Matters

For decades, the standard for big data was the Centralized Monolith. Organizations funneled all data into a single Data Warehouse or Data Lake, managed by a specialized “Data Team” that became a permanent bottleneck. As the number of data sources exploded and business domains became more complex, this model broke.

The Problem: The central data team lacks domain context (they don’t understand the business logic behind the data), while the domain teams (who understand the data) have no responsibility for its analytical value.

The Solution: Data Mesh, introduced by Zhamak Dehghani in 2019, flips the script. It applies the principles of Microservices and Product Thinking to data.

  • Historical Context: Evolved as a response to the failure of centralized “Data Lake” promises.
  • Real-World Impact: Used by companies like JPMorgan Chase, Zalando, and HelloFresh to scale data operations across hundreds of teams.
  • Why it matters: It’s the only architecture designed to scale with the complexity of a modern, diverse organization rather than just the volume of its data.

Core Concept Analysis

Data Mesh is built on four fundamental pillars. To understand Data Mesh, you must internalize these pillars not as “tools” but as shifts in organizational and technical behavior.

Pillar 1: Domain-Oriented Decentralized Data Ownership

Stop moving data to a central lake. Keep the responsibility for data with the people who create it.

CENTRALIZED (Old Way)              DATA MESH (New Way)
┌──────────────────────┐          ┌──────────┐      ┌──────────┐
│   Central Data Lake  │          │ Marketing│      │ Sales    │
│  (Bottleneck Team)   │          │ Domain   │      │ Domain   │
└──────────┬───────────┘          └────┬─────┘      └────┬─────┘
           │                           │                 │
    ┌──────┴──────┐             ┌──────▼──────┐   ┌──────▼──────┐
    │ Data Engs   │             │Data Product │   │Data Product │
    └─────────────┘             └─────────────┘   └─────────────┘

Pillar 2: Data as a Product

Analytical data is no longer a “byproduct” of an application; it is a Product with customers. It must be:

  • Discoverable: Users can find it in a catalog.
  • Addressable: It has a stable, unique endpoint.
  • Trustworthy: It has quality guarantees (SLAs).
  • Self-Describing: It includes schema and documentation.
  • Interoperable: It follows global standards (JSON, Parquet, etc.).

Pillar 3: Self-Serve Data Platform

Domain teams shouldn’t have to build their own database clusters. The Platform Team provides “Data Infrastructure as a Service.”

┌──────────────────────────────────────────────────────────┐
│               Self-Serve Data Platform                   │
├──────────────────────────────────────────────────────────┤
│ Provisioning | Storage | Compute | Identity | Lineage    │
└──────────────────────────────────────────────────────────┘
      ↑                 ↑                  ↑
┌───────────┐     ┌───────────┐      ┌───────────┐
│ Domain A  │     │ Domain B  │      │ Domain C  │
└───────────┘     └───────────┘      └───────────┘

Pillar 4: Federated Computational Governance

How do you prevent a “Data Swamp” in a decentralized world? You automate the rules. Governance isn’t a PDF; it’s code that runs in every data product (e.g., automated PII masking, schema validation).

Concept Summary Table

Concept Cluster What You Need to Internalize
Domain Ownership The people closest to the source are responsible for the analytical representation of that data.
Data Product Quanta The smallest unit of architecture in a mesh: includes code, data, metadata, and infrastructure.
Data Contracts The formal agreement between producer and consumer regarding schema, quality, and freshness.
Computational Policies Security, privacy, and compliance rules must be embedded in the platform, not checked manually.
Polyglot Interoperability Domains can use different tech, but they MUST talk to each other via standard protocols.

Deep Dive Reading by Concept

This section maps concepts to specific chapters. Read these alongside the projects.

The Foundation

Concept Book & Chapter
The 4 Pillars of Data Mesh Data Mesh by Zhamak Dehghani — Ch. 1-5
Analytical vs. Operational Data Data Mesh by Zhamak Dehghani — Ch. 8
Designing Data Systems Designing Data-Intensive Applications by Martin Kleppmann — Ch. 1

Data Product Design

Concept Book & Chapter
Data Product Architecture Data Mesh by Zhamak Dehghani — Ch. 9
Schema & Evolution Designing Data-Intensive Applications by Martin Kleppmann — Ch. 4
Data Modeling The Data Warehouse Toolkit by Ralph Kimball — Ch. 1-2

Essential Reading Order

  1. The Paradigm Shift (Week 1):
    • Data Mesh Ch. 1: “Data Mesh in a Nutshell”
    • Data Mesh Ch. 2: “Principle of Domain Ownership”
  2. The Architecture (Week 2):
    • Data Mesh Ch. 9: “Logical Architecture”
    • Designing Data-Intensive Applications Ch. 4: “Encoding and Evolution”

Project List

Projects are ordered from foundational concepts to advanced platform implementation.

Project 1: The Data Product Manifest (Defining the Quanta)

  • File: DATA_MESH_ARCHITECTURE_MASTERY.md
  • Main Programming Language: Python
  • Alternative Programming Languages: Go, TypeScript, Rust
  • Coolness Level: Level 2: Practical but Forgettable
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 1: Beginner
  • Knowledge Area: Metadata / Data Product Definition
  • Software or Tool: YAML, JSON Schema
  • Main Book: “Data Mesh” by Zhamak Dehghani (Ch. 9)

What you’ll build: A CLI tool that generates and validates a “Data Product Manifest.” This manifest is a single YAML file that defines everything about a data product: its schema (output ports), its owners, its documentation, its SLAs (freshness, availability), and its upstream dependencies.

Why it teaches Data Mesh: It forces you to think of data not as a table, but as an Architectural Quanta. You realize that “data” alone is useless without the context of who owns it, how often it updates, and what the schema looks like.

Core challenges you’ll face:

  • Defining “Output Ports” → How do you describe how someone actually accesses the data?
  • Schema Versioning → How do you represent that this manifest is for version 1.2.0?
  • Dependency Mapping → How do you link this product to the “Upstream” source products?

Key Concepts:

  • Data Product Quanta: Data Mesh (Ch. 9) - Zhamak Dehghani
  • Self-Describing Data: Designing Data-Intensive Applications (Ch. 4) - Martin Kleppmann

Difficulty: Beginner Time estimate: Weekend Prerequisites: Basic Python, understanding of YAML/JSON.

Real World Outcome

You will have a tool named mesh-cli that can initialize a new data product and validate its structure. This manifest becomes the “source of truth” for your mesh.

Example Output:

$ mesh-cli init --name "customer_orders" --domain "sales"
Created manifest.yaml

$ mesh-cli validate manifest.yaml
[OK] Name: customer_orders
[OK] Domain: sales
[OK] Output Ports: 2 (S3/Parquet, BigQuery)
[OK] Quality SLA: 99.9% freshness
[OK] Owner: sales-team@company.com
Validation Successful!

The Core Question You’re Answering

“What is the minimum amount of information required to make a piece of data a ‘Product’ rather than just a ‘Table’?”

Before you write any code, sit with this question. A table in a database is just bits. A product has a support team, a manual, and a guarantee.

Concepts You Must Understand First

Stop and research these before coding:

  1. Architectural Quantum
    • What components must be bundled together for a unit to be independently deployable?
    • Book Reference: “Data Mesh” Ch. 9 - Zhamak Dehghani
  2. Schema Definition Languages
    • Why is JSON Schema or Avro better than just a raw SQL CREATE TABLE statement for a mesh?
    • Book Reference: “Designing Data-Intensive Applications” Ch. 4 - Martin Kleppmann

Questions to Guide Your Design

Before implementing, think through these:

  1. Identity
    • How do you uniquely identify a data product across a global organization? (e.g., urn:mesh:sales:customer_orders)
  2. Standardization
    • What fields should be MANDATORY for every domain? What fields should be OPTIONAL?

Thinking Exercise

The “New Consumer” Test

Imagine you are a developer in the “Marketing” domain. You need order data from the “Sales” domain. Look at your manifest.yaml.

  • Can you tell who to Slack if the data is wrong?
  • Can you tell where to find the data files?
  • Do you know if the data is refreshed hourly or daily?

Questions while analyzing:

  • If any of these are “No”, your manifest is missing a critical “Product” attribute.

The Interview Questions They’ll Ask

Prepare to answer these:

  1. “What is a Data Product Quanta, and how does it differ from a microservice?”
  2. “Why is discoverability a first-class citizen in Data Mesh?”
  3. “How do you handle schema evolution in a decentralized system?”
  4. “What information belongs in a data contract vs. a data manifest?”
  5. “How would you automate the validation of 1000+ manifests?”

Hints in Layers

Hint 1: Start with a Schema Define a JSON Schema for your YAML manifest. Use a library like jsonschema in Python to validate it.

Hint 2: The Domain Namespace Ensure your tool enforces a naming convention: domain.subdomain.product_name.

Hint 3: Use Pydantic If using Python, use Pydantic to define the models. It makes validation and generating schemas much easier.

Hint 4: Documentation is Code Allow the manifest to point to a README.md file. A product without a manual isn’t a product.

Books That Will Help

Topic Book Chapter
Data Product Design “Data Mesh” by Zhamak Dehghani Ch. 3
Logical Architecture “Data Mesh” by Zhamak Dehghani Ch. 9
Schema Evolution “Designing Data-Intensive Applications” by Martin Kleppmann Ch. 4

Project 2: The Domain Publisher (Bridging Operational & Analytical)

  • File: DATA_MESH_ARCHITECTURE_MASTERY.md
  • Main Programming Language: Go
  • Alternative Programming Languages: Java, Python, Node.js
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 3. The “Service & Support” Model
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Event Streaming / CDC
  • Software or Tool: Postgres, Kafka (or Redpanda), Debezium
  • Main Book: “Designing Data-Intensive Applications” by Martin Kleppmann (Ch. 11)

What you’ll build: A service that implements the “Outbox Pattern.” It listens to changes in a Domain’s operational database (e.g., a “Sales” Postgres DB) and publishes those changes to a message bus (Kafka) in a format that conforms to a Data Product schema.

Why it teaches Data Mesh: It demonstrates the Separation of Concerns. The operational DB is for the application; the Kafka stream is the “Data Product” for the mesh. It teaches you how to decouple the two so that application changes don’t break the mesh.

Core challenges you’ll face:

  • Handling Schema Mismatch → The DB table might have 50 columns, but the Data Product only needs 10. How do you map them?
  • Guaranteed Delivery → How do you ensure that every DB transaction results in a Kafka message?
  • Metadata Injection → Adding “Mesh Metadata” (trace IDs, timestamps) to the stream.

Key Concepts:

  • Change Data Capture (CDC): Designing Data-Intensive Applications (Ch. 11)
  • The Outbox Pattern: Microservices Patterns (Ch. 3) - Chris Richardson

Difficulty: Intermediate Time estimate: 1-2 weeks Prerequisites: Basic Go, understanding of SQL and Message Queues.

Real World Outcome

You’ll have a running Postgres database and a Kafka broker. When you insert a row into the orders table, a cleanly formatted, versioned event appears in the mesh.sales.orders.v1 topic.

Example Output:

# In Postgres
INSERT INTO orders (id, amount, status) VALUES (1, 99.99, 'completed');

# In Kafka Consumer
{
  "metadata": {
    "product": "sales.orders",
    "version": "1.0.0",
    "timestamp": "2025-12-28T10:00:00Z"
  },
  "payload": {
    "order_id": 1,
    "total": 99.99,
    "state": "completed"
  }
}

The Core Question You’re Answering

“How do we expose analytical data without letting our internal database implementation details leak to the rest of the company?”

This is the “Operational vs. Analytical” divide. If you just give people a DB login, you are coupled. If you give them a curated stream, you are a Product Owner.

Concepts You Must Understand First

Stop and research these before coding:

  1. The Outbox Pattern
    • Why is it better than just sending a message from your application code?
  2. Log-Structured Streams
    • How does Kafka act as a “buffer” between domains?
    • Book Reference: “Designing Data-Intensive Applications” Ch. 11

Questions to Guide Your Design

Before implementing, think through these:

  1. Granularity
    • Should you publish every single update, or just “State Changes”?
  2. Versioning
    • If you add a column to the DB, does it automatically go to the stream? (Hint: No! That breaks the contract).

Thinking Exercise

The Breaking Change Trace

  1. You have a Data Product sales.orders.
  2. A developer changes the DB column status to order_status.
  3. If your publisher just passes through raw SQL rows, what happens to the consumers?
  4. How does a “Mapping Layer” in your publisher prevent this?

The Interview Questions They’ll Ask

  1. “What is CDC and why is it preferred over batch extracts for Data Mesh?”
  2. “How do you handle ‘Deleted’ records in a stream-based data product?”
  3. “What are the trade-offs of using an Event-Driven architecture for data sharing?”
  4. “How do you ensure ‘Exactly Once’ semantics in your publisher?”
  5. “How would you handle a schema change that IS a breaking change for consumers?”

Hints in Layers

Hint 1: Use a Listener In Postgres, you can use logical replication slots. If using Python, look at psycopg2 extras for logical replication.

Hint 2: Transform, Don’t Pass Through Create a function/class that specifically maps “Internal DB Row” to “Public Data Product Record.”

Hint 3: Schema Registry Use a Schema Registry (like Confluent’s) to ensure your Kafka messages match an Avro or Protobuf schema.

Hint 4: Versioning Include a v1 or v2 in the Kafka topic name itself. This is a common pattern for “Stable Output Ports.”

Books That Will Help

Topic Book Chapter
Stream Processing “Designing Data-Intensive Applications” by Martin Kleppmann Ch. 11
Event-Driven Design “Building Microservices” by Sam Newman Ch. 4

Project 3: The Data Contract Guardrail (Computational Governance)

  • File: DATA_MESH_ARCHITECTURE_MASTERY.md
  • Main Programming Language: Python
  • Alternative Programming Languages: Node.js (GitHub Actions), Go
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 3. The “Service & Support” Model
  • Difficulty: Level 3: Advanced
  • Knowledge Area: DevOps / Governance
  • Software or Tool: GitHub Actions, JSON Schema / Great Expectations
  • Main Book: “Data Mesh” by Zhamak Dehghani (Ch. 5)

What you’ll build: A “Governance Guardrail” that runs as part of a CI/CD pipeline. It fetches the “Data Contract” for a product and validates that the current code/schema changes don’t violate that contract. If a field is deleted or a type is changed in a way that breaks compatibility, the build fails.

Why it teaches Data Mesh: It moves governance from “Meetings and Checklists” to Computational Enforcement. This is the only way to maintain a mesh at scale without a central bureaucracy.

Core challenges you’ll face:

  • Backward Compatibility Checks → How do you programmatically determine if a schema change is “breaking”?
  • Global Policy Enforcement → Checking if a new data product includes mandatory PII tagging.
  • State Management → Where do you store the “current” contract to compare against the “new” one?

Key Concepts:

  • Federated Computational Governance: Data Mesh (Ch. 5)
  • Contract Testing: Building Microservices (Ch. 7)

Difficulty: Advanced Time estimate: 1-2 weeks Prerequisites: Understanding of CI/CD, JSON Schema, and Git.

Real World Outcome

A GitHub Action that comments on a Pull Request. If the dev tries to delete a user_id field that other teams rely on, the Action blocks the merge with a specific “Contract Violation” error.

Example Output:

✖ Build Failed: Contract Violation in 'sales.orders'
--------------------------------------------------
Error: Field 'total_amount' was removed.
Consumers Impacted: [Marketing-Analytics, Finance-Audit]
Action: You must increment the major version (v1 -> v2) 
        and provide a migration path to remove this field.

The Core Question You’re Answering

“How can we give teams total autonomy while ensuring they don’t accidentally break the rest of the company’s data?”

This is the fundamental tension of Data Mesh. The answer is automated, computational guardrails.

Concepts You Must Understand First

Stop and research these before coding:

  1. Semantic Versioning for Data
    • When is a change a “Minor” (compatible) vs. “Major” (breaking) change for data?
  2. PII (Personally Identifiable Information) Standards
    • What are the legal requirements for tagging data? (GDPR, CCPA).

Questions to Guide Your Design

Before implementing, think through these:

  1. The Policy Registry
    • Where do the “Global Rules” (like “all emails must be hashed”) live?
  2. Impact Analysis
    • How does the guardrail know WHO is consuming the data? (Hint: Lineage).

Thinking Exercise

The “Silent Break”

  1. A domain team changes a column price from Float to String.
  2. Technically, the data still loads.
  3. But the downstream “Finance Dashboard” starts throwing MathError.
  4. How could your Guardrail have caught this before it was merged?

The Interview Questions They’ll Ask

  1. “How do you enforce governance in a decentralized system without becoming a bottleneck?”
  2. “What is a ‘computational policy’ and give an example?”
  3. “How do you balance ‘Team Autonomy’ with ‘Global Interoperability’?”
  4. “How do you handle a scenario where a global policy changes? (e.g. New Privacy Law).”
  5. “What is the role of a ‘Platform Team’ in automated governance?”

Hints in Layers

Hint 1: Use diff Start by writing a script that takes two JSON schemas and identifies the differences.

Hint 2: Define “Breaking” Write a set of rules. For example: “Adding an optional field is OK. Removing any field is a BREAK. Changing type is a BREAK.”

Hint 3: Data Quality as Contract Include data quality checks (e.g., “Field X must never be NULL”) in the contract and validate them against sample data.

Hint 4: Open Policy Agent (OPA) For a truly “hardcore” version, use OPA and Rego to define your governance policies as code.

Books That Will Help

Topic Book Chapter
Federated Governance “Data Mesh” by Zhamak Dehghani Ch. 5
Testing Strategies “Building Microservices” by Sam Newman Ch. 7

Project 4: The Discovery Portal (Data Marketplace)

  • File: DATA_MESH_ARCHITECTURE_MASTERY.md
  • Main Programming Language: TypeScript (React/Next.js)
  • Alternative Programming Languages: Python (FastAPI + Jinja), Go
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 2. The “Micro-SaaS / Pro Tool”
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Web / Metadata Discovery
  • Software or Tool: SQLite/Postgres, Tailwind CSS
  • Main Book: “Data Mesh” by Zhamak Dehghani (Ch. 3)

What you’ll build: A “Marketplace” for data products. It aggregates the manifests from Project 1 and displays them in a searchable web interface. Users can search by domain, keyword, or SLA level. It should show the schema, the “Output Port” URLs (where to get the data), and the “Trustworthiness” score.

Why it teaches Data Mesh: It fulfills the Discoverable and Addressable requirements of a Data Product. You realize that in a decentralized world, the biggest problem isn’t “moving data,” it’s “finding data.”

Core challenges you’ll face:

  • Aggregation Strategy → How do you “crawl” or “collect” manifests from different domain Git repos?
  • Schema Visualization → How do you render complex nested schemas in a way that non-technical users understand?
  • Search & Filter → Implementing full-text search across documentation and metadata.

Key Concepts:

  • Discoverability: Data Mesh (Ch. 3)
  • Metadata Management: The Data Warehouse Toolkit (Ch. 1)

Difficulty: Intermediate Time estimate: 1-2 weeks Prerequisites: React/HTML/CSS, basic API design.

Real World Outcome

A browser-based portal where any developer can type “orders” and instantly see the sales.orders product, its current version, who owns it, and the SQL snippet to query it.

Example Output: (Web UI Mockup)

[ Search: orders ]
----------------------------------
Result: sales.orders (v1.2.0)
Domain: Sales
Owner: Jane Doe (@jane_sales)
SLA: 99.9% Uptime | Hourly Refresh
Schema: [id (int), amount (float), status (str)...]
Access: trino://mesh.internal:8080/sales/orders
----------------------------------

The Core Question You’re Answering

“If a developer needs a specific piece of information, how do they find it without having to ask someone on Slack?”

In a mesh, “Asking a human” is a failure of the architecture. The portal is the self-service interface.

Concepts You Must Understand First

Stop and research these before coding:

  1. Metadata vs. Data
    • What is the difference between “Technical Metadata” (schema) and “Business Metadata” (descriptions)?
  2. REST vs. GraphQL for Metadata APIs
    • Which one is better for exploring a deeply nested graph of data products?

Questions to Guide Your Design

Before implementing, think through these:

  1. The “Request Access” Flow
    • Once a user finds a product, how do they actually get permission to read it?
  2. Freshness Indicators
    • How can the portal show if the data is currently stale?

Thinking Exercise

The “Amazon” for Data

Think about the Amazon.com product page. It has:

  • Photos (Visual Representation)
  • Reviews (Trustworthiness)
  • “Frequently Bought Together” (Relationships)
  • Product Specs (Schema)

How would you translate “Customer Reviews” into a Data Mesh context? (Hint: Data Quality metrics or “Used by X teams”).

The Interview Questions They’ll Ask

  1. “How do you avoid creating a ‘Centralized Metadata Monolith’ while building a Discovery Portal?”
  2. “How do you handle ‘Stale’ documentation in a decentralized catalog?”
  3. “What are the most important metadata fields for a data consumer?”
  4. “How would you implement a ‘Rating’ or ‘Trust’ system for data products?”
  5. “Should the discovery portal also host the data, or just the pointers?”

Hints in Layers

Hint 1: The Scraper Write a simple script that walks through a directory of Git repos and looks for manifest.yaml files.

Hint 2: Unified API Build a simple FastAPI backend that serves a JSON list of all products to your React frontend.

Hint 3: Schema to Table Use a library like react-json-view or custom logic to render the JSON Schema into a clean HTML table.

Hint 4: Deep Linking Ensure every data product has a unique URL (e.g., /products/sales/orders) so teams can share links in documentation.

Books That Will Help

Topic Book Chapter
Product Thinking “Data Mesh” by Zhamak Dehghani Ch. 3
UX for Dev Tools “The Design of Everyday Things” by Don Norman Ch. 1

Project 5: Federated Query Engine (The Mesh Glue)

  • File: DATA_MESH_ARCHITECTURE_MASTERY.md
  • Main Programming Language: SQL / Java (Trino configuration)
  • Alternative Programming Languages: Python (DuckDB), Go (Presto)
  • Coolness Level: Level 5: Pure Magic
  • Business Potential: 4. The “Open Core” Infrastructure
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Distributed Systems / SQL Engines
  • Software or Tool: Trino (formerly PrestoSQL), MinIO (S3 clone), Postgres
  • Main Book: “Trino: The Definitive Guide” by Matt Fuller & Martin Traverso

What you’ll build: A federated query layer. You will set up Trino to connect to two different “Domain Ports”: one is an S3 bucket (Marketing) and one is a Postgres DB (Sales). You will then execute a single SQL query that joins data across these two physically separate locations as if they were one database.

Why it teaches Data Mesh: It realizes the Interoperable and Self-Serve pillars. It shows how the platform allows consumers to use the data however they want (via SQL) without the domain teams having to build custom APIs for every requester.

Core challenges you’ll face:

  • Catalog Configuration → Connecting Trino to multiple data sources with different credentials.
  • Performance Tuning → Understanding “Predicate Pushdown” (how Trino avoids downloading the whole internet to answer a query).
  • Identity Propagation → Ensuring the user’s identity is passed down to the underlying data sources.

Key Concepts:

  • Query Federation: Trino: The Definitive Guide (Ch. 1)
  • Compute/Storage Separation: Designing Data-Intensive Applications (Ch. 10)

Difficulty: Advanced Time estimate: 1-2 weeks Prerequisites: Strong SQL, Docker knowledge, basic Linux.

Real World Outcome

You can open a SQL terminal (like DBeaver or trino-cli) and run a query that calculates “Total Revenue by Marketing Campaign” by joining a CSV in S3 with an Orders table in Postgres.

Example Output:

SELECT 
  m.campaign_name, 
  SUM(s.amount) as revenue
FROM marketing.campaigns m  -- This is S3
JOIN sales.orders s        -- This is Postgres
ON m.customer_id = s.customer_id
GROUP BY 1;

-- Result:
-- [ "Black Friday", 14500.50 ]
-- [ "Summer Sale",  8200.00  ]

The Core Question You’re Answering

“How can we query data across the whole company without moving everything into a central data lake first?”

The “Federated Query Engine” is the alternative to the “Central Data Warehouse.” It provides a unified view without a unified storage layer.

Concepts You Must Understand First

Stop and research these before coding:

  1. Distributed Query Planning
    • How does a query engine split a single SQL statement into tasks for different machines?
  2. Catalogs vs. Schemas vs. Tables
    • How does Trino organize data from multiple sources?

Questions to Guide Your Design

Before implementing, think through these:

  1. The “Data Gravity” Problem
    • If Domain A has 1TB of data and Domain B has 1GB, where should the join happen?
  2. Access Control
    • If I have access to Trino, do I automatically have access to all underlying S3 buckets? (Hint: No, and your design should reflect that).

Thinking Exercise

The “Virtual” Join

Trace the path of a query:

  1. User sends SQL to Trino Coordinator.
  2. Coordinator looks at the “Marketing” catalog (MinIO) and “Sales” catalog (Postgres).
  3. It creates two plans.
  4. How does the data from Postgres “meet” the data from S3? Where does that meeting happen?

The Interview Questions They’ll Ask

  1. “What is query federation and why is it central to a Data Mesh?”
  2. “What are the trade-offs between ‘Federated Query’ and ‘ETL into a Data Warehouse’?”
  3. “Explain ‘Predicate Pushdown’ and why it matters for S3-based data products.”
  4. “How do you handle schema changes in a federated environment?”
  5. “What is the role of the ‘Metastore’ in a distributed query engine?”

Hints in Layers

Hint 1: Use Docker Compose Start by spinning up a trino container, a minio container, and a postgres container in one compose file.

Hint 2: Hive Metastore To query files in S3 (MinIO), Trino usually needs a “Hive Metastore” (HMS) to know which files map to which tables. Look into iceberg or delta-lake formats for an easier time.

Hint 3: Catalog Files Trino uses .properties files in /etc/trino/catalog/ to define connections. Create sales.properties and marketing.properties.

Hint 4: DuckDB Alternative If Trino is too heavy, try DuckDB. You can attach a Postgres DB and an S3 bucket in a single Python script and run SQL over them!

Books That Will Help

Topic Book Chapter
Distributed Queries “Trino: The Definitive Guide” by Fuller & Traverso Ch. 4
Data Integration “Designing Data-Intensive Applications” by Martin Kleppmann Ch. 10

Project 6: Automated PII Masker (Computational Policy Sidecar)

  • File: DATA_MESH_ARCHITECTURE_MASTERY.md
  • Main Programming Language: Python
  • Alternative Programming Languages: Go, Java
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 3. The “Service & Support” Model
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Security / Policy Enforcement
  • Software or Tool: Open Policy Agent (OPA), AWS Lambda or Flask
  • Main Book: “Data Mesh” by Zhamak Dehghani (Ch. 5)

What you’ll build: A “Policy Sidecar” that intercepts data requests. It checks the data product manifest (Project 1) for “PII Tags” (e.g., email, phone). If the requesting user doesn’t have the “Security Clearance” role, the sidecar automatically hashes or masks those specific columns before the data is returned.

Why it teaches Data Mesh: It demonstrates Federated Computational Governance. The policy is defined centrally (e.g., “Mask emails for non-admins”), but it is executed locally at the data product level.

Core challenges you’ll face:

  • Identity Mapping → Determining the roles of the user making the query.
  • Dynamic Masking → Applying logic based on column metadata (e.g., if tag=PII, then mask()).
  • Low Latency → Ensuring the policy check doesn’t slow down the data access.

Key Concepts:

  • Policy as Code: Data Mesh (Ch. 10)
  • Attribute-Based Access Control (ABAC): Cloud Native Infrastructure (Ch. 8)

Difficulty: Advanced Time estimate: 1-2 weeks Prerequisites: Basic understanding of JWTs/OAuth, Python, and JSON.

Real World Outcome

A proxy API. When an Admin calls GET /products/sales/orders, they see full emails. When a regular Analyst calls the same URL, they see j***@company.com.

Example Output:

// Admin Response
{ "id": 1, "email": "john.doe@gmail.com", "total": 100.0 }

// Analyst Response
{ "id": 1, "email": "j***@gmail.com", "total": 100.0 }

The Core Question You’re Answering

“How can we ensure data privacy without having to write custom masking code for every single data product?”

In a mesh, security is a platform capability, not a manual task for every domain.

Concepts You Must Understand First

Stop and research these before coding:

  1. Policy Engines (Rego/OPA)
    • How do you write a rule like “Allow if user.role == ‘admin’”?
  2. Column-Level Security
    • How is it different from “Row-Level Security”?

Questions to Guide Your Design

Before implementing, think through these:

  1. Performance
    • Should masking happen in the database or in the application layer?
  2. Audit Logs
    • How do you record why a specific piece of data was masked?

Thinking Exercise

Imagine a new law (like GDPR 2.0) requires all “Phone Numbers” to be masked.

  • In a traditional system, you’d have to update 500 different ETL jobs.
  • In your Data Mesh, you update one central policy in your Sidecar.
  • How does the manifest from Project 1 help this shift?

The Interview Questions They’ll Ask

  1. “What is Federated Computational Governance and why is it superior to manual audits?”
  2. “How do you handle ‘Policy Drift’ where different domains use different rules?”
  3. “Where should policy enforcement live: at the storage, engine, or application layer?”
  4. “Explain the difference between RBAC and ABAC in the context of a Data Mesh.”
  5. “How would you test that your masking policy is actually working?”

Hints in Layers

Hint 1: The Wrapper Create a simple Flask app that acts as a proxy to an underlying JSON file or DB.

Hint 2: Read the Manifest In your proxy code, load the manifest.yaml. Look for a tags list on each column.

Hint 3: Simple Logic Start with a hardcoded dictionary of rules: { "PII": lambda x: x[0] + "***" }.

Hint 4: Integrate OPA Use the python-opa library to send the request context to an OPA server and get back a “Mask/No Mask” decision.

Books That Will Help

Topic Book Chapter
Policy as Code “Cloud Native Infrastructure” by Justin Garrison Ch. 8
Data Governance “Data Mesh” by Zhamak Dehghani Ch. 5

Project 7: The Trust Scoreboard (SLO Monitoring)

  • File: DATA_MESH_ARCHITECTURE_MASTERY.md
  • Main Programming Language: Python
  • Alternative Programming Languages: Go, Node.js
  • Coolness Level: Level 2: Practical but Forgettable
  • Business Potential: 3. The “Service & Support” Model
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Monitoring / Reliability
  • Software or Tool: Prometheus / Grafana
  • Main Book: “Site Reliability Engineering” (SRE Book) by Google (Ch. 4)

What you’ll build: A service that monitors the health of data products. It reads the SLAs from the manifest (e.g., “Freshness: 60 mins”) and compares them against the actual data (e.g., MAX(created_at) in the database). It publishes a “Trust Score” for every data product.

Why it teaches Data Mesh: It validates the Trustworthy requirement. In a mesh, if consumers don’t trust the data, they will revert to building their own “shadow” data lakes. Trust is the currency of the mesh.

Core challenges you’ll face:

  • Defining “Freshness” → Is it when the record was created, or when it arrived in the mesh?
  • Sampling Strategy → How do you check quality on a 10TB table without scanning the whole thing?
  • Alert Fatigue → Ensuring domains only get notified when a real contract violation occurs.

Key Concepts:

  • SLIs, SLOs, and SLAs: SRE Book (Ch. 4)
  • Data Observability: Data Mesh (Ch. 11)

Difficulty: Intermediate Time estimate: 1 week Prerequisites: Basic SQL, Python, understanding of metrics/monitoring.

Real World Outcome

A dashboard where the “Sales” domain can see they have a “98% Trust Rating,” while the “Marketing” domain sees they are “failing” because their data is 2 hours late.

Example Output:

PRODUCT: sales.orders
SLA: Freshness < 1hr
ACTUAL: 12 mins
STATUS: [PASS] ✅

PRODUCT: marketing.leads
SLA: Freshness < 30 mins
ACTUAL: 145 mins
STATUS: [FAIL] ❌

Project 8: Self-Serve Infrastructure Provisioner (The Platform)

  • File: DATA_MESH_ARCHITECTURE_MASTERY.md
  • Main Programming Language: Go / Python (using Terraform/CDK)
  • Alternative Programming Languages: HCL (Terraform), TypeScript (Pulumi)
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 4. The “Open Core” Infrastructure
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Infrastructure as Code (IaC) / Platform Engineering
  • Software or Tool: Terraform, AWS (S3/IAM) or GCP
  • Main Book: “Terraform: Up & Running” by Yevgeniy Brikman

What you’ll build: A “Provisioner” for the Self-Serve Platform. A domain team submits their Manifest (Project 1), and this tool automatically creates the necessary cloud infrastructure (an S3 bucket for storage, an IAM role for the publisher, and a Glue database for metadata).

Why it teaches Data Mesh: It realizes the Self-Serve Data Platform pillar. It shows that the platform’s job is to reduce the “Lead Time” for a domain to publish their first data product from weeks to minutes.

Core challenges you’ll face:

  • Idempotency → Ensuring that running the tool twice doesn’t create two buckets.
  • Permission Mapping → Automatically generating the “Least Privilege” IAM policies based on the manifest.
  • State Management → Keeping track of which infrastructure belongs to which data product.

Key Concepts:

  • Platform as a Service (PaaS): Data Mesh (Ch. 4)
  • Infrastructure as Code: Terraform: Up & Running (Ch. 1)

Difficulty: Advanced Time estimate: 2 weeks Prerequisites: AWS/Cloud basics, Terraform or Boto3 knowledge.

Real World Outcome

A CLI command platform-deploy manifest.yaml. Five minutes later, the Sales team has a production-ready S3 bucket and a Service Account ready to start uploading data.

Project 9: The Identity Mesh (Entity Resolution)

  • File: DATA_MESH_ARCHITECTURE_MASTERY.md
  • Main Programming Language: Python (with Dedupe.io or RecordLinkage)
  • Alternative Programming Languages: Scala (Spark), Java
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 5. The “Industry Disruptor”
  • Difficulty: Level 4: Expert
  • Knowledge Area: Data Engineering / ML
  • Software or Tool: Python Record Linkage Toolkit
  • Main Book: “Data Mesh” by Zhamak Dehghani (Ch. 10 - Interoperability)

What you’ll build: A “Cross-Domain Joiner.” You take “User” data from the Sales domain and “Customer” data from the Marketing domain. Since they use different IDs, you implement a fuzzy matching algorithm (using name, email, and address) to create a “Linkage Table” that allows the mesh to join these two products accurately.

Why it teaches Data Mesh: It addresses the Interoperable requirement at a semantic level. In a decentralized system, “Identity” is the hardest problem. This project teaches you how to bridge the gap between domain-specific identifiers.

Core challenges you’ll face:

  • Fuzzy Matching → Handling “John Doe” vs. “J. Doe.”
  • Scalability → Matching 1M rows against 1M rows (O(n²) problem).
  • The “Source of Truth” Dilemma → Deciding which domain “owns” the master record.

Key Concepts:

  • Polyglot Interoperability: Data Mesh (Ch. 10)
  • Entity Resolution: Principles of Data Integration (Ch. 4)

Difficulty: Expert Time estimate: 2-4 weeks Prerequisites: Data science basics, advanced SQL, Python.

Real World Outcome

A new data product called identity.user_map. It contains a simple table: [sales_user_id, marketing_customer_id, confidence_score]. This allows the Federated Query Engine (Project 5) to join the two domains perfectly.

Project 10: Data Lineage Graph (The Map of the Mesh)

  • File: DATA_MESH_ARCHITECTURE_MASTERY.md
  • Main Programming Language: Python / JavaScript (D3.js)
  • Alternative Programming Languages: Go, Java
  • Coolness Level: Level 5: Pure Magic
  • Business Potential: 3. The “Service & Support” Model
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Graph Theory / Observability
  • Software or Tool: OpenLineage, Marquez, or Neo4j
  • Main Book: “Data Mesh” by Zhamak Dehghani (Ch. 11)

What you’ll build: A visualization tool that maps the dependencies between data products. It parses the “Upstream Dependencies” field from the manifests and the query logs from the Federated Engine to draw a live graph of how data flows across the organization.

Why it teaches Data Mesh: It provides Mesh Observability. Without lineage, a mesh becomes an unmanageable web of “spaghetti dependencies.” It teaches you how to maintain visibility in a decentralized system.

Core challenges you’ll face:

  • Parsing SQL → Extracting table names from raw SQL strings to find “who is querying what.”
  • Graph Layouts → Rendering 100+ nodes in a way that isn’t just a “hairball.”
  • Time-Traveling Lineage → Seeing how the dependencies looked last month vs. today.

Key Concepts:

  • Data Lineage: Data Mesh (Ch. 11)
  • Graph Theory Basics: Algorithms (Ch. 4) - Sedgewick

Difficulty: Advanced Time estimate: 2 weeks Prerequisites: Basic Graph theory, Python, frontend visualization basics.

Project Comparison Table

Project Difficulty Time Depth of Understanding Fun Factor
1. Manifest CLI Beginner Weekend High (Pillars 1-2) 3/5
2. Domain Publisher Intermediate 1-2 Weeks Medium (Operational Bridge) 4/5
3. Contract Guardrail Advanced 1 Week Very High (Governance) 4/5
4. Discovery Portal Intermediate 2 Weeks Medium (Product Thinking) 5/5
5. Federated Engine Advanced 1-2 Weeks High (Platform/Interop) 5/5
6. PII Sidecar Advanced 1 Week High (Security) 4/5
7. SLO Monitor Intermediate 1 Week Medium (Trust) 3/5
8. Infra Provisioner Advanced 2 Weeks Medium (Platform) 3/5
9. Identity Mesh Expert 1 Month Very High (Complexity) 4/5
10. Lineage Graph Advanced 2 Weeks High (Observability) 5/5

Recommendation

If you are a Backend Engineer: Start with Project 2 (The Publisher). It connects the world you know (Databases) to the world of Data Mesh (Streams/Products).

If you are a Data Engineer: Start with Project 5 (Federated Engine). It will change how you think about “data movement” forever.

If you are a DevOps/SRE: Start with Project 3 (Contract Guardrail). It’s the most practical way to see how “Governance as Code” works.

Final Overall Project: The “Instant Mesh” Framework

The Ultimate Challenge

Build a “Data Mesh in a Box” framework.

Requirements:

  • A single configuration command that spins up a local Mesh environment (Trino, Kafka, MinIO).
  • A base library (in Python or Go) that domain teams can import to automatically handle manifest validation, PII masking, and CDC publishing.
  • A central “Control Plane” dashboard that shows the Discovery Portal, the Lineage Graph, and the SLO Trust Scores in one view.

This project combines all previous 10 projects into a single, cohesive platform. If you complete this, you aren’t just a data engineer; you are a Data Mesh Architect.

Summary

This learning path covers Data Mesh Architecture through 10 hands-on projects. Here’s the complete list:

# Project Name Main Language Difficulty Time Estimate
1 The Data Product Manifest Python Beginner Weekend
2 The Domain Publisher Go Intermediate 1-2 weeks
3 The Contract Guardrail Python Advanced 1 week
4 The Discovery Portal TypeScript Intermediate 2 weeks
5 Federated Query Engine SQL/Java Advanced 1-2 weeks
6 Automated PII Masker Python Advanced 1 week
7 The SLO Monitor Python Intermediate 1 week
8 Infra Provisioner Go/HCL Advanced 2 weeks
9 The Identity Mesh Python Expert 1 month
10 Data Lineage Graph Python/JS Advanced 2 weeks

For beginners: Start with projects #1, #2, #4. Focus on understanding the “Product” mindset. For intermediate: Focus on #3, #5, #7. Master the balance between autonomy and governance. For advanced: Focus on #6, #8, #9, #10. Build the high-level platform and solve the “Hard” problems like identity and lineage.

Expected Outcomes

After completing these projects, you will:

  • Design Data Products: Understand how to package data, code, and metadata into a deployable unit.
  • Implement Governance as Code: Automate PII masking and contract validation using CI/CD.
  • Master Query Federation: Join data across disparate systems without moving it.
  • Build Self-Serve Platforms: Provision cloud infrastructure automatically for domain teams.
  • Think in Mesh: Shift from “centralized pipeline” thinking to “decentralized ecosystem” thinking.

You’ll have built 10 working projects that demonstrate deep understanding of Data Mesh from first principles.