Project 16: Full Mail Server Stack (Capstone)

Build a complete mail system: inbound SMTP, outbound delivery, authentication, spam filtering, storage, and IMAP access.

Quick Reference

Attribute	Value
Difficulty	Master
Time Estimate	2-3 months
Language	C (core) + Python (tooling) (Alternatives: Go, Rust)
Prerequisites	All previous projects, strong systems skills
Key Topics	SMTP server/client, queueing, auth, storage, IMAP

1. Learning Objectives

Design and implement a complete mail pipeline.
Build a durable queue with retry schedules.
Integrate SPF/DKIM/DMARC validation and DKIM signing.
Provide IMAP access to stored mail.

2. Theoretical Foundation

2.1 Core Concepts

MTA vs MDA vs MUA: Separate responsibilities for transfer, delivery, and access.
Queueing and retries: SMTP is best-effort; delivery requires retry with backoff.
Authentication stack: SPF and DKIM results inform DMARC policy.
Storage formats: Maildir for durability and IMAP compatibility.

2.2 Why This Matters

This project ties the entire email ecosystem together. You will understand how production mail systems handle reliability, security, and scale.

2.3 Historical Context / Background

Email servers evolved over decades, accumulating features to combat spam, abuse, and reliability challenges. Modern MTAs are complex for a reason.

2.4 Common Misconceptions

Misconception: SMTP acceptance means delivery. Reality: it only means queue acceptance.
Misconception: Auth checks are optional. Reality: they are required for deliverability.

3. Project Specification

3.1 What You Will Build

A full mail server stack that can receive and store inbound mail, send outbound mail with DKIM signing, enforce authentication policies, run basic spam filtering, and provide IMAP retrieval.

3.2 Functional Requirements

Inbound SMTP server with proper state machine and TLS.
Outbound SMTP client with queue and retry schedule.
SPF, DKIM verification and DMARC policy enforcement for inbound.
DKIM signing for outbound.
Maildir storage and IMAP server for retrieval.
Config-driven policies and logging.

3.3 Non-Functional Requirements

Reliability: Messages are durable once accepted.
Security: TLS for submission and IMAP, strict parsing.
Observability: Logs and metrics for queue and delivery.

3.4 Example Usage / Output

$ ./mailserver start --domain example.com --config /etc/mailserver/config.yml
[INIT] SMTP inbound :25
[INIT] SMTP submission :587
[INIT] IMAP :993
[READY] Mail server running

[SMTP:25] MAIL FROM:<a@gmail.com> RCPT TO:<user@example.com>
[AUTH] SPF: PASS DKIM: PASS DMARC: PASS
[STORE] Maildir saved: /var/mail/user/new/1700000000.123

3.5 Real World Outcome

You can host a working email domain with inbound and outbound mail flow, validate authentication, and access mail via IMAP clients.

4. Solution Architecture

4.1 High-Level Design

Inbound SMTP
  -> Auth Checks (SPF/DKIM/DMARC)
  -> Spam Filter
  -> Maildir Storage
  -> IMAP Server

Outbound Queue
  -> DKIM Signer
  -> SMTP Client
  -> Retry Scheduler

4.2 Key Components

Component	Responsibility	Key Decisions
Inbound SMTP	Accept mail	strict state machine
Auth Engine	SPF/DKIM/DMARC	reuse project logic
Queue	Store outbound	disk-based queue
IMAP	Retrieve mail	minimal IMAP commands

4.3 Data Structures

typedef struct {
    char id[64];
    char sender[256];
    char recipient[256];
    char path[512];
    int attempts;
    time_t next_retry;
} queue_item_t;

4.4 Algorithm Overview

Key Algorithm: Queue Retry

On delivery failure, set next_retry = now + backoff.
Increment attempts.
Move to dead letter after max attempts.

Complexity Analysis:

Time: O(n) to scan queue
Space: O(n) for queued items

5. Implementation Guide

5.1 Development Environment Setup

# Build core components
make

5.2 Project Structure

mailserver/
├── smtp_inbound/
├── smtp_outbound/
├── auth/
├── queue/
├── maildir/
├── imap/
└── config/

5.3 The Core Question You’re Answering

“How do all pieces of the email stack work together to deliver a message reliably and securely?”

5.4 Concepts You Must Understand First

Stop and research these before coding:

SMTP server and client flows
Queue durability and retry policies
SPF/DKIM/DMARC integration
Maildir and IMAP basics

5.5 Questions to Guide Your Design

What retry schedule will you use for outbound mail?
Where will you enforce DMARC policy?
How will you prevent resource exhaustion?

5.6 Thinking Exercise

If a message fails due to temporary DNS failure, how many retries and what intervals should be used to avoid excessive delays?

5.7 The Interview Questions They’ll Ask

“Why does an MTA need a queue?”
“How does DMARC affect inbound acceptance?”
“How would you handle a spike in inbound connections?”

5.8 Hints in Layers

Hint 1: Build inbound first

Use the mini SMTP server as a base.

Hint 2: Add auth checks next

Integrate SPF/DKIM/DMARC before storage.

Hint 3: Add outbound queue last

Start with immediate delivery, then add retries.

5.9 Books That Will Help

Topic	Book	Chapter
SMTP and sockets	TLPI	Ch. 56-63
Mail systems	Sendmail docs	architecture sections
Reliability	Release It!	Ch. 5

5.10 Implementation Phases

Phase 1: Foundation (2-3 weeks)

Goals:

Inbound SMTP and Maildir storage

Tasks:

Implement SMTP server with TLS.
Store messages in Maildir.

Checkpoint: Accept and store messages.

Phase 2: Core Functionality (3-4 weeks)

Goals:

Authentication and outbound sending

Tasks:

Integrate SPF/DKIM/DMARC checks.
Implement outbound SMTP client with DKIM signing.

Checkpoint: Send mail that passes authentication.

Phase 3: Polish and Edge Cases (3-4 weeks)

Goals:

Queue, IMAP, and monitoring

Tasks:

Build disk-backed queue with retries.
Implement minimal IMAP server.
Add logging and metrics.

Checkpoint: Full end-to-end mail flow.

5.11 Key Implementation Decisions

Decision	Options	Recommendation	Rationale
Queue storage	SQLite vs files	files	simpler and fast
IMAP scope	minimal vs full	minimal core commands	scope control
TLS handling	openssl vs library	openssl or language TLS	stable and known

6. Testing Strategy

6.1 Test Categories

Category	Purpose	Examples
Unit Tests	Auth checks	SPF/DKIM/DMARC logic
Integration Tests	SMTP flows	send and receive mail
End-to-End Tests	IMAP retrieval	fetch stored mail

6.2 Critical Test Cases

Inbound mail stored before 250 response.
Outbound mail retries on 4xx.
DMARC fail causes rejection/quarantine.

6.3 Test Data

Test domain with SPF/DKIM/DMARC configured

7. Common Pitfalls and Debugging

7.1 Frequent Mistakes

Pitfall	Symptom	Solution
No durable queue	lost mail	write to disk before ack
Weak auth checks	spoofed mail	enforce DMARC policy
IMAP state bugs	missing messages	consistent Maildir handling

7.2 Debugging Strategies

Add per-session logs with message IDs.
Trace message flow from acceptance to delivery.

7.3 Performance Traps

Serial processing of inbound connections. Use concurrency.

8. Extensions and Challenges

8.1 Beginner Extensions

Add basic web admin UI.
Add quota limits per mailbox.

8.2 Intermediate Extensions

Add spam filtering integration.
Add MTA-STS and TLS-RPT support.

8.3 Advanced Extensions

Add multi-tenant virtual domains.
Implement ARC for forwarded mail.

9. Real-World Connections

9.1 Industry Applications

Mail providers use similar pipelines for reliability and security.
Enterprises rely on queues and auth checks for compliance.

Postfix: https://www.postfix.org/
Dovecot: https://www.dovecot.org/

9.3 Interview Relevance

End-to-end mail architecture is a strong capstone for systems interviews.

10. Resources

10.1 Essential Reading

RFC 5321 - SMTP
RFC 5322 - Message format
RFC 6376 - DKIM
RFC 7208 - SPF
RFC 7489 - DMARC

10.2 Video Resources

Mail server architecture talks

10.3 Tools and Documentation

openssl s_client for TLS
swaks for SMTP testing

11. Self-Assessment Checklist

11.1 Understanding

I can explain MTA/MDA/MUA roles
I understand queue and retry policies
I can explain auth enforcement

11.2 Implementation

Inbound and outbound mail flow works
IMAP retrieval works
Auth checks are enforced

11.3 Growth

I can operate a mail server safely
I can debug end-to-end mail delivery issues

12. Submission / Completion Criteria

Minimum Viable Completion:

Inbound SMTP server with Maildir storage

Full Completion:

Outbound delivery with DKIM signing and retries

Excellence (Going Above and Beyond):

Full IMAP support and monitoring
Multi-domain hosting and advanced policies

This guide was generated from EMAIL_SYSTEMS_DEEP_DIVE_PROJECTS.md. For the complete learning path, see the parent directory.

Project 16: Full Mail Server Stack (Capstone)

Quick Reference

1. Learning Objectives

2. Theoretical Foundation

2.1 Core Concepts

2.2 Why This Matters

2.3 Historical Context / Background

2.4 Common Misconceptions

3. Project Specification

3.1 What You Will Build

3.2 Functional Requirements

3.3 Non-Functional Requirements

3.4 Example Usage / Output

3.5 Real World Outcome

4. Solution Architecture

4.1 High-Level Design

4.2 Key Components

4.3 Data Structures

4.4 Algorithm Overview

5. Implementation Guide

5.1 Development Environment Setup

5.2 Project Structure

5.3 The Core Question You’re Answering

5.4 Concepts You Must Understand First

5.5 Questions to Guide Your Design

5.6 Thinking Exercise

5.7 The Interview Questions They’ll Ask

5.8 Hints in Layers

5.9 Books That Will Help

5.10 Implementation Phases

Phase 1: Foundation (2-3 weeks)

Phase 2: Core Functionality (3-4 weeks)

Phase 3: Polish and Edge Cases (3-4 weeks)

5.11 Key Implementation Decisions

6. Testing Strategy

6.1 Test Categories

6.2 Critical Test Cases

6.3 Test Data

7. Common Pitfalls and Debugging

7.1 Frequent Mistakes

7.2 Debugging Strategies

7.3 Performance Traps

8. Extensions and Challenges

8.1 Beginner Extensions

8.2 Intermediate Extensions

8.3 Advanced Extensions

9. Real-World Connections

9.1 Industry Applications

9.2 Related Open Source Projects

9.3 Interview Relevance

10. Resources

10.1 Essential Reading

10.2 Video Resources

10.3 Tools and Documentation

10.4 Related Projects in This Series

11. Self-Assessment Checklist

11.1 Understanding

11.2 Implementation

11.3 Growth

12. Submission / Completion Criteria