Project 1: Raw SMTP Client from Scratch

Build a CLI that speaks SMTP over TCP and can deliver a real email with STARTTLS and AUTH.

Quick Reference

Attribute	Value
Difficulty	Intermediate
Time Estimate	1-2 weeks
Language	Python (Alternatives: C, Go, Rust)
Prerequisites	TCP sockets, basic TLS concepts, RFC reading
Key Topics	SMTP state machine, reply codes, STARTTLS, SMTP AUTH

1. Learning Objectives

By completing this project, you will:

Implement a strict SMTP state machine with correct command sequencing.
Parse multiline SMTP responses and handle retry logic for 4xx results.
Upgrade a plain TCP connection to TLS via STARTTLS mid-session.
Implement at least one SMTP AUTH mechanism with safe credential handling.

2. Theoretical Foundation

2.1 Core Concepts

SMTP as a state machine: SMTP is not a single request. It is a sequence of states that must occur in order. Each response code drives the next valid command.
Reply codes and semantics: 2xx is success, 3xx is intermediate, 4xx is temporary failure, 5xx is permanent failure. Client behavior depends on which you see.
STARTTLS upgrade: The client starts in cleartext, then negotiates TLS after the server advertises STARTTLS. The connection is re-wrapped, and the SMTP session effectively restarts with EHLO.
AUTH mechanisms: AUTH PLAIN and LOGIN are common. They are simple but must be used only after TLS, otherwise credentials are exposed.

2.2 Why This Matters

You cannot debug deliverability or authentication failures if you do not understand the raw protocol. Building this client makes the protocol observable and teaches you to reason about failures without relying on libraries.

2.3 Historical Context / Background

SMTP dates to the early 1980s and predates ubiquitous encryption. STARTTLS and AUTH were bolted on later, which is why the protocol has this upgrade-in-place behavior instead of a clean secure transport from the start.

2.4 Common Misconceptions

Misconception: SMTP is a single request. Reality: It is a stateful dialogue.
Misconception: STARTTLS is optional. Reality: Many servers require it on submission ports.
Misconception: You can send AUTH before EHLO. Reality: Many servers reject that sequence.

3. Project Specification

3.1 What You Will Build

A CLI tool that connects to a mail server, negotiates capabilities, performs STARTTLS, authenticates, and sends a complete RFC 5322 message with correct dot-stuffing.

3.2 Functional Requirements

TCP connection to a specified host and port.
Capability discovery via EHLO, with multiline response parsing.
STARTTLS upgrade if advertised, followed by EHLO again.
AUTH with PLAIN and/or LOGIN.
Message send with MAIL FROM, RCPT TO, DATA, and termination by dot line.
Transcript logging with redacted credentials.

3.3 Non-Functional Requirements

Performance: Should handle a typical send in under 2 seconds on a local network.
Reliability: Retry on 4xx for RCPT TO or DATA with backoff.
Usability: Clear error messages that include server reply code and text.

3.4 Example Usage / Output

$ ./smtp-client --server smtp.gmail.com --port 587 \
  --from you@gmail.com --to friend@example.com \
  --subject "Test from raw SMTP" --body "Hello"

[CONNECT] smtp.gmail.com:587
[RECV] 220 smtp.gmail.com ESMTP ready
[SEND] EHLO myclient.local
[RECV] 250-smtp.gmail.com Hello
[RECV] 250-STARTTLS
[RECV] 250 AUTH PLAIN LOGIN
[SEND] STARTTLS
[RECV] 220 Ready to start TLS
[TLS] Negotiated TLS 1.3
[SEND] EHLO myclient.local
[RECV] 250 AUTH PLAIN LOGIN
[SEND] AUTH PLAIN <redacted>
[RECV] 235 Authentication successful
[SEND] MAIL FROM:<you@gmail.com>
[RECV] 250 OK
[SEND] RCPT TO:<friend@example.com>
[RECV] 250 OK
[SEND] DATA
[RECV] 354 End data with <CR><LF>.<CR><LF>
[SEND] <headers and body>
[RECV] 250 Message accepted
[SEND] QUIT
[RECV] 221 Bye
SUCCESS: Email sent

3.5 Real World Outcome

You can point the tool at any SMTP submission server, and it will reliably send a message while showing the exact protocol exchange. You will see a full transcript and can explain any rejection by reading the reply codes.

4. Solution Architecture

4.1 High-Level Design

CLI
  -> TCP Socket
    -> SMTP Session
      -> Capability Parser
      -> TLS Upgrader
      -> Authenticator
      -> Message Sender

4.2 Key Components

Component	Responsibility	Key Decisions
Session	Track SMTP state and enforce ordering	Explicit state enum per RFC 5321
Parser	Decode replies, handle multiline	Parse by code and hyphen continuation
TLS	Upgrade socket to TLS	Re-EHLO after STARTTLS
Auth	Implement AUTH mechanisms	Base64 encode, redact logs

4.3 Data Structures

class Reply:
    def __init__(self, code: int, lines: list[str]):
        self.code = code
        self.lines = lines

class SessionState:
    DISCONNECTED = 0
    CONNECTED = 1
    EHLO_DONE = 2
    TLS_ACTIVE = 3
    AUTH_DONE = 4
    MAIL_TXN = 5

4.4 Algorithm Overview

Key Algorithm: Reply Parsing

Read lines until a line with “" terminates.
For each line, extract the code and text.
Return Reply(code, lines).

Complexity Analysis:

Time: O(n) for n lines
Space: O(n) for reply storage

5. Implementation Guide

5.1 Development Environment Setup

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip

5.2 Project Structure

smtp-client/
├── smtp_client.py
├── smtp_state.py
├── tls_wrap.py
├── auth.py
└── README.md

5.3 The Core Question You’re Answering

“What exactly happens on the wire when an email is sent, and how do servers enforce correctness?”

5.4 Concepts You Must Understand First

Stop and research these before coding:

SMTP Reply Codes
- 2xx vs 4xx vs 5xx
- Why 354 is required before DATA
- RFC 5321 Sections 4.2, 4.5
STARTTLS Upgrade
- Why EHLO must be repeated
- How TLS wraps an existing socket
- Security impact of sending AUTH in cleartext
SMTP AUTH
- Base64 encoding vs encryption
- AUTH PLAIN vs LOGIN
- When servers advertise AUTH
Dot-stuffing
- Why lines starting with ‘.’ are escaped
- Terminating DATA with a lone dot

5.5 Questions to Guide Your Design

How will you detect a multiline reply and when to stop reading?
What should happen if STARTTLS is not advertised?
How will you keep credentials out of logs?
What are the correct state transitions for EHLO, MAIL, RCPT, DATA?

5.6 Thinking Exercise

Trace this SMTP conversation and mark the states:

S: 220 example.com
C: EHLO client
S: 250-example.com
S: 250-STARTTLS
S: 250 AUTH PLAIN LOGIN
C: STARTTLS
S: 220 Ready
C: EHLO client
S: 250 AUTH PLAIN LOGIN

Where do you reset state? Where does TLS begin? Which capability list do you trust?

5.7 The Interview Questions They’ll Ask

“Why must you send EHLO again after STARTTLS?”
“What is the difference between 4xx and 5xx in SMTP?”
“What is dot-stuffing and why does it exist?”
“Why is AUTH PLAIN safe only over TLS?”

5.8 Hints in Layers

Hint 1: Start with a transcript logger

Store every send and receive line to debug state errors.

Hint 2: Parse replies by code

Check if the 4th character is ‘-‘ to keep reading.

Hint 3: Build a state enum

Fail fast if a command is sent in the wrong state.

5.9 Books That Will Help

Topic	Book	Chapter
SMTP protocol	RFC 5321	Sections 2-4
Socket I/O	TCP/IP Illustrated Vol 1	Ch. 7
TLS basics	Serious Cryptography	Ch. 12
Network programming	TLPI	Ch. 56-61

5.10 Implementation Phases

Phase 1: Foundation (2-3 days)

Goals:

Connect via TCP
Read greeting and send EHLO

Tasks:

Implement socket connect and line reads.
Parse multiline replies.

Checkpoint: Can print capabilities from EHLO.

Phase 2: Core Functionality (4-5 days)

Goals:

STARTTLS upgrade
AUTH and message send

Tasks:

Upgrade socket to TLS and re-EHLO.
Implement AUTH PLAIN or LOGIN.
Send MAIL FROM, RCPT TO, DATA.

Checkpoint: Successfully send a test email.

Phase 3: Polish and Edge Cases (2-3 days)

Goals:

Handle 4xx retries
Robust logging and errors

Tasks:

Add backoff for 4xx on RCPT.
Redact logs.

Checkpoint: Clear error for any non-2xx response.

5.11 Key Implementation Decisions

Decision	Options	Recommendation	Rationale
Reply parsing	Regex vs manual	Manual parser	More control and clarity
AUTH method	PLAIN vs LOGIN	PLAIN over TLS	Simpler and standard
TLS library	stdlib vs external	stdlib	Fewer dependencies

6. Testing Strategy

6.1 Test Categories

Category	Purpose	Examples
Unit Tests	Parse replies	Multiline reply parsing
Integration Tests	Talk to local SMTP	Send to local test server
Edge Case Tests	Error handling	421, 450, 550 responses

6.2 Critical Test Cases

Multiline EHLO: ensure parser stops on final line.
STARTTLS failure: abort if 454 or missing capability.
Dot-stuffing: lines starting with ‘.’ are escaped.

6.3 Test Data

250-example.com
250-STARTTLS
250 AUTH PLAIN LOGIN

7. Common Pitfalls and Debugging

7.1 Frequent Mistakes

Pitfall	Symptom	Solution
Skipping EHLO after STARTTLS	AUTH fails	Re-EHLO after TLS
Incorrect reply parsing	Hang or premature stop	Check hyphen rule
Sending bare LF	Server rejects	Always use CRLF

7.2 Debugging Strategies

Compare transcript to RFC examples.
Log state transitions and reject invalid steps.

7.3 Performance Traps

Excessive reconnects for retries. Use backoff and reuse the session when possible.

8. Extensions and Challenges

8.1 Beginner Extensions

Add support for AUTH LOGIN.
Allow multiple RCPT TO recipients.

8.2 Intermediate Extensions

Implement SMTP UTF8 and 8BITMIME support.
Add message builder with attachments (MIME).

8.3 Advanced Extensions

Implement SMTP pipelining and concurrency.
Add SASL mechanisms like CRAM-MD5.

9. Real-World Connections

9.1 Industry Applications

Mail clients use the same sequence for submission and sending.
Monitoring tools rely on low-level SMTP checks to validate deliverability.

swaks: https://www.jetmore.org/john/code/swaks/ - SMTP testing tool
msmtp: https://marlam.de/msmtp/ - Lightweight SMTP client

9.3 Interview Relevance

SMTP state machines and TLS upgrades are common in systems and networking interviews.

10. Resources

10.1 Essential Reading

RFC 5321 - SMTP protocol details and reply codes
RFC 5322 - Message format and headers

10.2 Video Resources

SMTP protocol walkthroughs (protocol debugging)

10.3 Tools and Documentation

openssl s_client: inspect STARTTLS behavior
swaks: compare your transcript to a known tool

MX Record Resolver for routing logic
Mini SMTP Server for inbound handling

11. Self-Assessment Checklist

11.1 Understanding

I can explain SMTP reply codes and their meaning
I can explain STARTTLS and why EHLO is repeated
I can explain dot-stuffing

11.2 Implementation

Sends an email to a real provider
Handles multiline replies correctly
Credentials are not logged

11.3 Growth

I can debug SMTP failures without third-party libraries
I can explain how SMTP AUTH works

12. Submission / Completion Criteria

Minimum Viable Completion:

Can connect, EHLO, and send mail to a local test server
Correctly parse multiline replies
Handles DATA termination with dot-stuffing

Full Completion:

Sends mail via STARTTLS with AUTH
Handles 4xx retries gracefully

Excellence (Going Above and Beyond):

Implements pipelining and multiple recipients
Provides a clean transcript and JSON output mode

This guide was generated from EMAIL_SYSTEMS_DEEP_DIVE_PROJECTS.md. For the complete learning path, see the parent directory.

Project 1: Raw SMTP Client from Scratch

Quick Reference

1. Learning Objectives

2. Theoretical Foundation

2.1 Core Concepts

2.2 Why This Matters

2.3 Historical Context / Background

2.4 Common Misconceptions

3. Project Specification

3.1 What You Will Build

3.2 Functional Requirements

3.3 Non-Functional Requirements

3.4 Example Usage / Output

3.5 Real World Outcome

4. Solution Architecture

4.1 High-Level Design

4.2 Key Components

4.3 Data Structures

4.4 Algorithm Overview

5. Implementation Guide

5.1 Development Environment Setup

5.2 Project Structure

5.3 The Core Question You’re Answering

5.4 Concepts You Must Understand First

5.5 Questions to Guide Your Design

5.6 Thinking Exercise

5.7 The Interview Questions They’ll Ask

5.8 Hints in Layers

5.9 Books That Will Help

5.10 Implementation Phases

Phase 1: Foundation (2-3 days)

Phase 2: Core Functionality (4-5 days)

Phase 3: Polish and Edge Cases (2-3 days)

5.11 Key Implementation Decisions

6. Testing Strategy

6.1 Test Categories

6.2 Critical Test Cases

6.3 Test Data

7. Common Pitfalls and Debugging

7.1 Frequent Mistakes

7.2 Debugging Strategies

7.3 Performance Traps

8. Extensions and Challenges

8.1 Beginner Extensions

8.2 Intermediate Extensions

8.3 Advanced Extensions

9. Real-World Connections

9.1 Industry Applications

9.2 Related Open Source Projects

9.3 Interview Relevance

10. Resources

10.1 Essential Reading

10.2 Video Resources

10.3 Tools and Documentation

10.4 Related Projects in This Series

11. Self-Assessment Checklist

11.1 Understanding

11.2 Implementation

11.3 Growth

12. Submission / Completion Criteria