Project 12: Open Relay Tester

Build a tool that tests whether an SMTP server is misconfigured as an open relay.

Quick Reference

Attribute Value
Difficulty Intermediate
Time Estimate 1 week
Language Python (Alternatives: Go, Rust)
Prerequisites SMTP client basics, protocol states
Key Topics Relay rules, SMTP test cases, safe testing

1. Learning Objectives

  1. Implement safe SMTP relay tests.
  2. Evaluate server responses to different MAIL FROM/RCPT TO combinations.
  3. Report relay status with evidence.
  4. Avoid abusive behavior while testing.

2. Theoretical Foundation

2.1 Core Concepts

  • Open relay: An SMTP server that accepts mail from unauthenticated clients to external domains.
  • Relay policy: Most servers allow relay only for local senders or authenticated users.
  • Test matrix: Using external and internal sender/recipient combos.

2.2 Why This Matters

Open relays are abused for spam and quickly blacklisted. Testing prevents misconfiguration and security incidents.

2.3 Historical Context / Background

Early SMTP servers were open by default. Modern relaying requires explicit authentication and policy enforcement.

2.4 Common Misconceptions

  • Misconception: Accepting RCPT TO means relay allowed. Reality: final acceptance might still fail at DATA.
  • Misconception: One test case is enough. Reality: multiple combinations are needed.

3. Project Specification

3.1 What You Will Build

A CLI that performs a set of safe relay tests, reports whether the server allows unauthorized relaying, and provides evidence.

3.2 Functional Requirements

  1. Connect to SMTP server and EHLO.
  2. Run test cases with different MAIL FROM and RCPT TO pairs.
  3. Stop before DATA if possible to avoid sending mail.
  4. Report relay status and server replies.

3.3 Non-Functional Requirements

  • Safety: Never send actual email content.
  • Reliability: Handle 4xx/5xx responses correctly.
  • Usability: Summarize results clearly.

3.4 Example Usage / Output

$ ./relay-test mail.example.com
Test 1: external -> external
  RCPT TO rejected: 550 Relaying denied
Test 2: local -> external
  RCPT TO rejected: 550 Relaying denied
Result: Not an open relay

3.5 Real World Outcome

You can prove whether an SMTP server is safe from open relay abuse and provide a test transcript for audits.

4. Solution Architecture

4.1 High-Level Design

SMTP Client
  -> Test Matrix
  -> Response Analyzer
  -> Report Generator

4.2 Key Components

Component Responsibility Key Decisions
Client SMTP dialogue Use minimal commands
Matrix Test cases external/internal combinations
Analyzer Interpret responses track allow/deny
Reporter Output results include evidence

4.3 Data Structures

class TestCase:
    def __init__(self, mail_from, rcpt_to):
        self.mail_from = mail_from
        self.rcpt_to = rcpt_to

4.4 Algorithm Overview

Key Algorithm: Relay Test

  1. Connect and EHLO.
  2. Send MAIL FROM and RCPT TO.
  3. Record response codes.
  4. Stop before DATA when possible.

Complexity Analysis:

  • Time: O(t) for t test cases
  • Space: O(1)

5. Implementation Guide

5.1 Development Environment Setup

python -m venv .venv
source .venv/bin/activate

5.2 Project Structure

relay-test/
├── smtp_client.py
├── tests.py
└── report.py

5.3 The Core Question You’re Answering

“Does this server allow unauthenticated relaying to external domains?”

5.4 Concepts You Must Understand First

Stop and research these before coding:

  1. SMTP command order
  2. Relay policy concepts
  3. Response code meanings
  4. Safe testing practices

5.5 Questions to Guide Your Design

  1. Which combinations count as relay abuse?
  2. When should you stop the test to avoid sending mail?
  3. How will you handle servers that accept RCPT but reject DATA?

5.6 Thinking Exercise

If RCPT TO is accepted for an external recipient but DATA is rejected, is the server an open relay? Why or why not?

5.7 The Interview Questions They’ll Ask

  1. “What is an open relay and why is it dangerous?”
  2. “Why test multiple sender/recipient combinations?”
  3. “What SMTP response codes indicate relay denial?”

5.8 Hints in Layers

Hint 1: Build a simple SMTP client

  • Reuse code from Project 1.

Hint 2: Stop before DATA

  • Avoid sending actual content.

Hint 3: Store evidence

  • Record responses for each test.

5.9 Books That Will Help

Topic Book Chapter
SMTP RFC 5321 Section 4
Mail security Practical Email Security relay section

5.10 Implementation Phases

Phase 1: Foundation (2-3 days)

Goals:

  • SMTP connection and EHLO

Tasks:

  1. Connect and parse responses.
  2. Send MAIL FROM and RCPT TO.

Checkpoint: Successful command exchange.

Phase 2: Core Functionality (2-3 days)

Goals:

  • Test matrix and reporting

Tasks:

  1. Implement multiple test cases.
  2. Collect results and print report.

Checkpoint: Determine relay status.

Phase 3: Polish and Edge Cases (1-2 days)

Goals:

  • Handle DATA rejection

Tasks:

  1. Optionally test DATA without sending content.
  2. Add safety checks and warnings.

Checkpoint: Safe tests with clear results.

5.11 Key Implementation Decisions

Decision Options Recommendation Rationale
Test cases minimal vs comprehensive comprehensive Avoid false negatives
DATA step skip vs minimal skip if possible Safety
Output text vs JSON both audits and automation

6. Testing Strategy

6.1 Test Categories

Category Purpose Examples
Unit Tests Response parsing 550 relay denied
Integration Tests Test server local postfix config
Edge Case Tests RCPT accepted DATA denied

6.2 Critical Test Cases

  1. External to external should be denied.
  2. Local to external should be denied if not authenticated.
  3. Local to local should be allowed.

6.3 Test Data

MAIL FROM:<test@external.com>
RCPT TO:<victim@external.com>

7. Common Pitfalls and Debugging

7.1 Frequent Mistakes

Pitfall Symptom Solution
Sending DATA unintended email stop after RCPT
Misreading 250 false positives check DATA acceptance too
Using real domains unintended impact use test domains

7.2 Debugging Strategies

  • Log every response code.
  • Compare with known relay tests (swaks).

7.3 Performance Traps

  • Too many tests without delay. Add pacing.

8. Extensions and Challenges

8.1 Beginner Extensions

  • Add STARTTLS support.
  • Add authenticated relay test.

8.2 Intermediate Extensions

  • Implement VRFY/EXPN checks.
  • Add config for custom test addresses.

8.3 Advanced Extensions

  • Integrate with audit report generation.
  • Add scanning for multiple servers.

9. Real-World Connections

9.1 Industry Applications

  • Security audits for mail servers.
  • Compliance checks for relay configuration.
  • swaks: https://www.jetmore.org/john/code/swaks/
  • smtp-cli: https://github.com/mludvig/smtp-cli

9.3 Interview Relevance

  • SMTP security and relay policy are common in email infrastructure roles.

10. Resources

10.1 Essential Reading

  • RFC 5321 - SMTP relay behavior

10.2 Video Resources

  • SMTP relay testing walkthroughs

10.3 Tools and Documentation

  • swaks relay testing examples

11. Self-Assessment Checklist

11.1 Understanding

  • I can define open relay
  • I understand SMTP relay policy
  • I can interpret relay test responses

11.2 Implementation

  • Runs a safe relay test matrix
  • Produces clear evidence and results
  • Avoids sending real mail

11.3 Growth

  • I can audit relay configuration confidently
  • I can explain relay security risks

12. Submission / Completion Criteria

Minimum Viable Completion:

  • Test external to external relay and report result

Full Completion:

  • Run a test matrix and report evidence

Excellence (Going Above and Beyond):

  • Add authenticated relay tests and audit output

This guide was generated from EMAIL_SYSTEMS_DEEP_DIVE_PROJECTS.md. For the complete learning path, see the parent directory.