← Back to all projects

LEARN MQTT DEEP DIVE

Learn MQTT: From Zero to IoT Master

Goal: Deeply understand the MQTT protocol—from its lightweight publish-subscribe model and binary format to building your own broker, client libraries, and networked IoT applications from scratch in C.

Why Learn MQTT?

MQTT (Message Queuing Telemetry Transport) is the de-facto standard for IoT and machine-to-machine (M2M) communication. It’s designed for high-latency, low-bandwidth, and unreliable networks, making it perfect for connecting everything from tiny embedded sensors to massive cloud platforms.

After completing these projects, you will:

  • Understand every byte in an MQTT packet.
  • Implement the complete publish-subscribe flow.
  • Build a concurrent MQTT broker capable of handling thousands of clients.
  • Grasp the trade-offs between different Quality of Service (QoS) levels.
  • Know how to secure MQTT communication.
  • Appreciate why MQTT has become dominant in the IoT space and understand its alternatives.

Core Concept Analysis

The MQTT Landscape: Pub/Sub Architecture

At its heart, MQTT is a publish-subscribe protocol that decouples clients. A client publishing a message doesn’t know who will receive it. A client receiving a message doesn’t know who sent it. The Broker is the central hub responsible for this routing.

                                  ┌──────────────────┐
                                  │   MQTT Broker    │
                                  │ (Central Server) │
                                  └──────────────────┘
                                      ▲          │
                                      │          ▼
                           Publish to │          │ Subscribe to
                          "temp/sensor1"         │ "temp/sensor1"
                                      │          │
                       ┌──────────────┴──────────┴──────────────┐
                       │                                        │
                       ▼                                        │
             ┌─────────────────┐                       ┌──────────────────┐
             │  Client A       │                       │  Client B        │
             │ (Temperature    ├──────────────────────►│ (Dashboard App)  │
             │  Sensor)        │                       └──────────────────┘
             └─────────────────┘
                   Publishes a message
                    (e.g., "22.5°C")

Key Concepts Explained

  1. MQTT Packet Format: All communication is done through a simple binary format with three parts:
    • Fixed Header (2-5 bytes): Identifies the packet type (e.g., CONNECT, PUBLISH), flags, and remaining length.
    • Variable Header: Contains additional information specific to the packet type (e.g., Packet ID for QoS > 0).
    • Payload: The actual application data (e.g., a sensor reading, a command). This is opaque to the broker.

  2. Topics: A UTF-8 string that acts as a “channel” for messages. They are hierarchical, using / as a separator (e.g., home/living_room/temperature). Clients can use wildcards (+ for single-level, # for multi-level) to subscribe to multiple topics at once.

  3. Quality of Service (QoS): MQTT provides three levels of delivery guarantees:
    • QoS 0 (At most once): Fire-and-forget. The message is sent once with no acknowledgment.
    • QoS 1 (At least once): The message is guaranteed to be delivered at least once. The receiver must send a PUBACK packet. The sender will re-transmit if it doesn’t receive one.
    • QoS 2 (Exactly once): The most reliable, but also slowest, level. It uses a four-part handshake to ensure the message is delivered exactly once, without duplicates.
  4. The Broker: The central server that manages client connections, subscriptions, and message routing. It is responsible for authenticating clients, receiving all published messages, and forwarding them to clients subscribed to matching topics.

Competitors & Alternatives

Protocol Model Strengths Weaknesses Best For
MQTT Pub/Sub Lightweight, low power, simple, great for unreliable networks. Limited built-in routing logic, broker is a single point of failure. IoT, mobile notifications, sensor networks.
AMQP Pub/Sub, Queues Feature-rich, complex routing, transactions, strong guarantees. Heavyweight protocol, more complex to set up and use. Enterprise messaging, financial systems.
CoAP Req/Resp Designed for constrained devices, UDP-based, like HTTP for IoT. Not inherently pub/sub (requires an extension), less mature ecosystem. Resource-constrained devices needing request-response interaction.
WebSockets Full-duplex Low-latency, bidirectional communication over a single TCP connection. Not a messaging protocol itself, just a transport layer. No built-in pub/sub. Real-time web applications (chat, live dashboards).
ZeroMQ Library (not protocol) Brokerless, many patterns (req/rep, pub/sub, pipeline), very fast. More of a developer toolkit than a defined protocol, interoperability is up to you. High-performance computing, financial trading, inter-process communication.

Project List

The following 10 projects will guide you from the binary fundamentals of MQTT to building a complete, networked IoT application in C.

Project 1: MQTT Packet Parser

  • File: LEARN_MQTT_DEEP_DIVE.md
  • Main Programming Language: C
  • Alternative Programming Languages: Python, Rust, Go
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Network Protocol / Binary Parsing
  • Software or Tool: A hex editor or Wireshark to view packet captures.
  • Main Book: “TCP/IP Illustrated, Volume 1: The Protocols” by W. Richard Stevens

What you’ll build: A command-line tool that reads a raw MQTT packet from a file or stdin and decodes its headers and payload into a human-readable format. It will explain each flag and field, similar to what wireshark does.

Why it teaches MQTT: You cannot master a protocol until you understand its on-the-wire representation. This project forces you to read the MQTT specification and translate it directly into code. After this, you will be able to visualize the protocol’s structure in your head.

Core challenges you’ll face:

  • Parsing the Fixed Header → maps to decoding packet types and flags from the first byte.
  • Decoding the “Remaining Length” field → maps to implementing the variable-length encoding scheme, where a single integer can span 1-4 bytes.
  • Handling different Variable Headers → maps to understanding that each packet type has a different variable header structure.
  • Extracting Payload data → maps to correctly calculating the start of the payload based on the fixed and variable header lengths.

Key Concepts:

  • MQTT Control Packet Format: MQTT v3.1.1 Specification, Section 2.
  • Variable-Length “Remaining Length” Encoding: MQTT v3.1.1 Specification, Section 2.2.3.
  • Bitwise Operations in C: “The C Programming Language” by Kernighan & Ritchie, Chapter 2.

Difficulty: Intermediate Time estimate: 1-2 weeks Prerequisites: Solid C programming skills, including pointers, structs, and bitwise operations.

Real world outcome:

$ cat publish_packet.bin | ./mqtt_parser
--- MQTT Packet Analysis ---
[FIXED HEADER]
Packet Type: PUBLISH (3)
Flags: DUP=0, QoS=1, Retain=0
Remaining Length: 36

[VARIABLE HEADER]
Topic Name: /sensors/living_room/temp
Packet Identifier: 101

[PAYLOAD]
Payload Size: 22 bytes
Payload (ASCII): {"value": 24.5, "unit": "C"}

Implementation Hints:

Start with the first byte of the fixed header.

// This is an explanation, not code to copy.
// Read the first byte.
// The upper 4 bits are the packet type.
unsigned char first_byte = read_byte();
unsigned char packet_type = first_byte >> 4;
// The lower 4 bits are the flags.
unsigned char flags = first_byte & 0x0F;

// Then, implement the "Remaining Length" decoding loop.
int multiplier = 1;
int value = 0;
unsigned char encoded_byte;
do {
    encoded_byte = read_byte();
    value += (encoded_byte & 127) * multiplier;
    multiplier *= 128;
} while ((encoded_byte & 128) != 0);

// Now you know the total size of the rest of the packet.
// You can read it into a buffer and parse the variable header and payload based on the packet_type.

Questions to guide you:

  1. How do you know if a packet is a PUBLISH, SUBSCRIBE, or something else just from the first 4 bits?
  2. The “Remaining Length” can be up to 268,435,455. How does its variable-length encoding scheme work, and why is it designed that way?
  3. For a PUBLISH packet, how do you determine if the Packet Identifier field is present in the variable header? (Hint: check the QoS level).

Learning milestones:

  1. Parse Fixed Header correctly → You understand how every MQTT message is identified.
  2. Implement Remaining Length decoding → You’ve mastered the protocol’s core variable-length data mechanism.
  3. Parse CONNECT and PUBLISH packets → You can decode the two most common packet types.
  4. The tool handles any valid packet → You have a solid foundation for building a client or broker.

Project 2: Simple CONNECT Client

  • File: LEARN_MQTT_DEEP_DIVE.md
  • Main Programming Language: C
  • Alternative Programming Languages: Python, Go
  • Coolness Level: Level 2: Practical but Forgettable
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Network Programming / Sockets
  • Software or Tool: A public MQTT broker like test.mosquitto.org.
  • Main Book: “The Linux Programming Interface” by Michael Kerrisk (for sockets).

What you’ll build: A C program that connects to a public MQTT broker, sends a valid CONNECT packet, and successfully receives and parses the CONNACK (Connection Acknowledged) packet.

Why it teaches MQTT: This is the “hello, world” of MQTT. It teaches the fundamental lifecycle of a client: establishing a network connection, performing the MQTT handshake, and verifying a successful connection before doing anything else.

Core challenges you’ll face:

  • TCP Socket Programming → maps to using socket(), connect() to establish a network connection to the broker.
  • Constructing a CONNECT Packet → maps to building your first valid MQTT packet from scratch, including the variable header and payload (Client ID, flags, etc.).
  • Sending and Receiving Data → maps to using send() and recv() to transmit the packet and read the response.
  • Parsing the CONNACK Packet → maps to validating that the connection was successful by checking the return code in the response.

Key Concepts:

  • TCP Client Socket Workflow: “TCP/IP Sockets in C” by Donahoo & Calvert, Chapter 4.
  • CONNECT Packet Structure: MQTT v3.1.1 Specification, Section 3.1.
  • CONNACK Packet Structure: MQTT v3.1.1 Specification, Section 3.2.

Difficulty: Intermediate Time estimate: Weekend Prerequisites: Project 1 (or a solid understanding of the packet structure), basic C socket programming.

Real world outcome:

$ ./mqtt_connect test.mosquitto.org 1883 my-test-client-123
Connecting to test.mosquitto.org:1883...
Sending CONNECT packet...
Received CONNACK packet.
Return Code: 0 (Connection Accepted)
Success!

Implementation Hints:

The process is sequential:

  1. Use getaddrinfo() to resolve the broker’s hostname to an IP address.
  2. Create a TCP socket with socket().
  3. Connect to the broker with connect().
  4. Construct the CONNECT packet in a memory buffer. Remember to include:
    • The protocol name (“MQTT”) and version number in the variable header.
    • Connect flags (e.g., Clean Session).
    • A unique Client ID in the payload.
  5. Calculate the “Remaining Length” and complete the fixed header.
  6. Use send() to write the buffer to the socket.
  7. Use recv() to read the broker’s response (it will be exactly 4 bytes for a CONNACK).
  8. Parse the CONNACK packet and check the return code.

A key question: What is the “Clean Session” flag, and what are the implications of setting it to 0 vs. 1? This is fundamental to how the broker handles client state.

Learning milestones:

  1. A TCP connection is established → You understand the underlying network layer.
  2. The broker receives your CONNECT packet without error → You can craft a valid MQTT handshake.
  3. You successfully parse the CONNACK → You can handle the server’s response and confirm success.
  4. The program handles connection errors gracefully → You are building robust network code.

Project 3: Command-Line PUBLISH and SUBSCRIBE Client

  • File: LEARN_MQTT_DEEP_DIVE.md
  • Main Programming Language: C
  • Alternative Programming Languages: Go, Python
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 2. The “Micro-SaaS / Pro Tool”
  • Difficulty: Level 2: Intermediate
  • Knowledge Area: Network Client / CLI Tool Design
  • Software or Tool: test.mosquitto.org
  • Main Book: “The C Programming Language” by Kernighan & Ritchie (for CLI parsing).

What you’ll build: A single command-line tool that can operate in two modes: pub or sub. In pub mode, it connects, sends a PUBLISH packet with a given topic and message, and exits. In sub mode, it connects, sends a SUBSCRIBE packet for a given topic, and then enters a loop to listen for and print any incoming PUBLISH packets.

Why it teaches MQTT: This project implements the two most important operations in MQTT. You’ll build the client-side logic for both sending and receiving data, and in doing so, create a tool that is genuinely useful for testing and debugging—a simplified version of mosquitto_pub and mosquitto_sub.

Core challenges you’ll face:

  • Crafting PUBLISH and SUBSCRIBE Packets → maps to learning the specific variable header and payload formats for these packet types.
  • Handling Asynchronous Input → maps to in subscribe mode, using select() or poll() to wait for data to arrive on the socket without blocking indefinitely.
  • State Management → maps to handling SUBACK packets to confirm a subscription before starting to listen for messages.
  • Parsing Incoming PUBLISH Packets → maps to re-using your parsing logic from Project 1 to decode messages received from the broker.

Key Concepts:

  • PUBLISH Packet Structure: MQTT v3.1.1 Specification, Section 3.3.
  • SUBSCRIBE Packet Structure: MQTT v3.1.1 Specification, Section 3.8.
  • I/O Multiplexing: “The Linux Programming Interface” by Michael Kerrisk, Chapter 63 (select() and poll()).

Difficulty: Intermediate Time estimate: 1-2 weeks Prerequisites: Project 2.

Real world outcome:

Terminal 1:

$ ./mqtt_client sub -h test.mosquitto.org -t "gemini/test/topic"
Subscribed to gemini/test/topic. Waiting for messages...
[gemini/test/topic] Hello from Terminal 2!
[gemini/test/topic] This is a test.

Terminal 2:

$ ./mqtt_client pub -h test.mosquitto.org -t "gemini/test/topic" -m "Hello from Terminal 2!"
Message published.
$ ./mqtt_client pub -h test.mosquitto.org -t "gemini/test/topic" -m "This is a test."
Message published.

Implementation Hints:

For the subscriber, the main loop will look something like this:

  1. Connect and send SUBSCRIBE packet.
  2. Wait for and validate the SUBACK packet.
  3. Enter a while(1) loop.
  4. Inside the loop, use select() to wait for the socket to become readable.
  5. When it is, recv() the data. The first byte will tell you the packet type.
  6. If it’s a PUBLISH packet, parse it and print the topic and payload.
  7. Handle potential disconnections and PINGREQ/PINGRESP keep-alive messages.

A key challenge is parsing a stream of multiple packets. After you handle one packet, your buffer might still contain the start of the next one. You need to manage your read buffer carefully.

Learning milestones:

  1. You can publish a message that another client receives → You’ve successfully implemented the publisher side of the protocol.
  2. You can subscribe and receive a message → You’ve successfully implemented the receiver side.
  3. The subscriber can handle multiple messages over time → You understand how to manage a persistent client connection.
  4. The tool is a functional replacement for basic mosquitto_pub/sub → You’ve built a genuinely useful piece of software.

Project 4: Concurrent In-Memory Broker

  • File: LEARN_MQTT_DEEP_DIVE.md
  • Main Programming Language: C
  • Alternative Programming Languages: Go, Rust
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 4. The “Open Core” Infrastructure
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Network Server / Concurrency
  • Software or Tool: Your client from Project 3 for testing.
  • Main Book: “Operating Systems: Three Easy Pieces” by Arpaci-Dusseau (for concurrency concepts).

What you’ll build: Your first server. A C program that acts as an MQTT broker. It will listen for TCP connections, handle multiple clients concurrently using select() or epoll(), and manage subscriptions in memory. When it receives a PUBLISH packet, it will forward it to all subscribed clients.

Why it teaches MQTT: This is the heart of the system. Building a broker forces you to think about the entire system at once: managing state for many clients, implementing the routing logic, and handling the protocol from the server’s perspective. It’s a significant step up from writing a client.

Core challenges you’ll face:

  • Server-side Socket Programming → maps to using socket(), bind(), listen(), and accept() to manage incoming connections.
  • Concurrent Client Management → maps to using I/O multiplexing (select, poll, epoll) to handle reading from dozens or hundreds of clients without using threads.
  • Subscription Management → maps to designing a data structure (e.g., a hash map of topics to a list of client sockets) to track which client is subscribed to which topic.
  • Message Routing Logic → maps to iterating through your subscription data structure after a PUBLISH is received and forwarding the message to the correct clients.

Key Concepts:

  • I/O Multiplexing Models: “Understanding and Using epoll” (LibDev). epoll is preferred for high-performance servers on Linux.
  • Server Architectures: “The C10K Problem” by Dan Kegel.
  • Data Structures for Routing: A trie is an excellent choice for handling topic wildcards (+, #).

Difficulty: Advanced Time estimate: 1 month+ Prerequisites: Project 3, strong understanding of C, data structures, and memory management.

Real world outcome: You run your broker:

$ ./my_broker
MQTT Broker started on port 1883. Waiting for clients...
[INFO] New client connected: fd=5
[INFO] Client 5 subscribed to 'home/lights'
[INFO] New client connected: fd=6
[INFO] Client 6 subscribed to 'home/lights'
[INFO] Client 5 published to 'home/lights'. Forwarding to 2 subscribers.

Your clients from Project 3 can now connect to localhost and communicate through your broker.

Implementation Hints:

Your main data structures might look like this:

// This is a conceptual guide.
struct Client {
    int socket_fd;
    // ... other state like keep-alive timer
};

struct Subscription {
    char* topic_filter;
    struct Client* client;
};

// Main state for the broker
struct Broker {
    int listening_socket;
    struct Client clients[MAX_CLIENTS];
    // A more advanced structure for subscriptions, e.g., a trie
    struct Subscription subscriptions[MAX_SUBS];
};

The main loop will be centered around epoll_wait() (on Linux) or select(). When a socket is ready:

  1. If it’s the listening socket, accept() a new client, add them to your epoll set and your client list.
  2. If it’s a client socket, recv() the data.
  3. Parse the packet(s).
  4. If it’s a CONNECT, send CONNACK.
  5. If it’s a SUBSCRIBE, add an entry to your subscription table and send SUBACK.
  6. If it’s a PUBLISH, look up the topic in your subscription table and send() the packet to every matching client.
  7. If recv() returns 0 or -1, the client has disconnected; clean up their state and subscriptions.

Learning milestones:

  1. Multiple clients can connect and subscribe → You have a working concurrent server architecture.
  2. A published message is correctly forwarded to one subscriber → Your core routing logic works.
  3. A published message is correctly forwarded to multiple subscribers → Your one-to-many logic is solid.
  4. The broker runs for hours without crashing or leaking memory → You’ve built a stable, long-running network service.

Project 5: Implementing QoS 1

  • File: LEARN_MQTT_DEEP_DIVE.md
  • Main Programming Language: C
  • Alternative Programming Languages: Rust, Go
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 4. The “Open Core” Infrastructure
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Reliable Messaging / State Management
  • Software or Tool: Your broker and client from previous projects.
  • Main Book: “Designing Data-Intensive Applications” by Martin Kleppmann (for concepts on message delivery guarantees).

What you’ll build: Extend both your client and broker to support QoS 1 (“at least once”) delivery. This involves adding Packet Identifiers to PUBLISH packets and implementing the PUBACK handshake.

Why it teaches MQTT: This is the single most important feature that elevates MQTT from a simple messaging protocol to a reliable one. Implementing QoS 1 forces you to manage stateful, multi-step message exchanges, handle acknowledgments, and implement retry logic for unreliable networks.

Core challenges you’ll face:

  • Packet Identifier Management → maps to generating unique Packet IDs for in-flight messages and tracking them.
  • Stateful Message Tracking (Client) → maps to storing a copy of a QoS 1 PUBLISH packet until a PUBACK with the matching Packet ID is received.
  • Retry Logic (Client) → maps to implementing a timeout mechanism to re-send the PUBLISH packet if a PUBACK is not received in time.
  • Sending PUBACKs (Broker) → maps to when the broker receives a QoS 1 PUBLISH, it must send a PUBACK packet back to the original publisher.

Key Concepts:

  • QoS 1 PUBLISH Flow: MQTT v3.1.1 Specification, Section 4.3.2.
  • PUBACK Packet: MQTT v3.1.1 Specification, Section 3.4.
  • State Machines in C: A common way to implement protocol handshakes.

Difficulty: Advanced Time estimate: 2-3 weeks Prerequisites: Project 4.

Real world outcome: You can now run your client with a -q 1 flag. Even if you artificially add network delay or packet loss (e.g., with tc), your tool can guarantee the message gets through.

Client Side:

$ ./mqtt_client pub -h localhost -t "test" -m "reliable message" -q 1
Sent PUBLISH (QoS 1, Packet ID 123). Waiting for PUBACK...
Received PUBACK for Packet ID 123. Message delivered reliably.

Broker Logs:

[INFO] Client 5 published to 'test' (QoS 1, Packet ID 123).
[INFO] Sending PUBACK for Packet ID 123 to Client 5.
[INFO] Forwarding PUBLISH to 1 subscriber.

Implementation Hints:

On the client, you’ll need a new data structure to track in-flight messages:

struct InFlightMessage {
    uint16_t packet_id;
    unsigned char* packet_data;
    size_t packet_len;
    time_t sent_time;
    int retry_count;
};

When you send a QoS 1 PUBLISH, add it to an array of InFlightMessage. In your select/epoll loop, you’ll need to:

  1. Check for incoming PUBACK packets. If one arrives, find the matching message in your InFlightMessage array and remove it.
  2. Periodically iterate through the InFlightMessage array. If any message has sent_time older than your timeout, re-send it and increment retry_count.

On the broker, when you receive a PUBLISH with QoS > 0, you must:

  1. Extract the Packet ID from the variable header.
  2. Send a PUBACK packet containing the same Packet ID back to the sender.
  3. Then, proceed with routing the message to subscribers.

Learning milestones:

  1. Broker responds to QoS 1 PUBLISH with a PUBACK → You’ve implemented the server side of the handshake.
  2. Client correctly identifies and processes the PUBACK → You’ve implemented the client side of the handshake.
  3. Client re-transmits a PUBLISH if no PUBACK is received → Your retry logic works, providing the reliability guarantee.
  4. The system correctly handles multiple in-flight messages at once → Your state management is robust.

Project 6: Broker with Topic Wildcard Support

  • File: LEARN_MQTT_DEEP_DIVE.md
  • Main Programming Language: C
  • Alternative Programming Languages: C++, Rust
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 4. The “Open Core” Infrastructure
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Data Structures / Algorithms
  • Software or Tool: Your existing broker.
  • Main Book: “Algorithms, Fourth Edition” by Sedgewick & Wayne (for Tries).

What you’ll build: Enhance your broker’s subscription logic to correctly handle MQTT’s two topic wildcards: + (single-level) and # (multi-level).

Why it teaches MQTT: Wildcards are a power-user feature of MQTT that make designing complex systems much easier. Implementing them requires moving beyond simple hash maps for subscription management and using a more appropriate data structure, like a Trie, that understands the hierarchical nature of topics.

Core challenges you’ll face:

  • Choosing the Right Data Structure → maps to realizing a simple hash map on the topic string is insufficient and selecting a tree-like structure.
  • Implementing a Trie → maps to building a Trie data structure where each node represents a level in the topic hierarchy (e.g., ‘home’, ‘living_room’).
  • Wildcard Matching Logic → maps to traversing the Trie to find all subscriptions, including those with + and #, that match a given published topic.
  • Efficiently Storing Subscribers → maps to attaching a list of client sockets to the nodes in the Trie that represent the end of a subscription filter.

Key Concepts:

  • Topic Wildcards (+, #): MQTT v3.1.1 Specification, Section 4.7.
  • Trie (Prefix Tree) Data Structure: A classic data structure for string-based lookups and prefix matching.
  • Recursive Tree Traversal: The natural way to implement the matching logic on a Trie.

Difficulty: Advanced Time estimate: 2 weeks Prerequisites: Project 4.

Real world outcome: A client can subscribe to sensors/+/temperature and receive messages published to sensors/living_room/temperature and sensors/bedroom/temperature, but not sensors/garage/humidity.

Broker Logs:

[INFO] Client 7 subscribed to 'sensors/+/temperature'.
[INFO] Client 8 published to 'sensors/living_room/temperature'.
[INFO] Match found for wildcard subscription 'sensors/+/temperature'.
[INFO] Forwarding message to Client 7.

Implementation Hints: A Trie is the canonical data structure for this problem.

// Conceptual Trie Node
struct TrieNode {
    char* level_name; // e.g., "sensors"
    bool is_end_of_subscription;
    struct ClientList* subscribers; // List of clients subscribed to this exact topic
    struct TrieNode* children; // Linked list or hash map of child nodes
};

When a client subscribes to home/+/lights:

  1. You traverse the Trie: home -> + -> lights.
  2. You create nodes if they don’t exist.
  3. At the final lights node, you add the client to its list of subscribers.

When a message is published to home/living_room/lights:

  1. You start a recursive traversal of the Trie.
  2. At each level, you check for three possible matches: a. A node with the exact level name (e.g., living_room). b. A node named +. c. A node named # (which matches the rest of the topic).
  3. You gather all subscribers from all matching paths.

Learning milestones:

  1. A subscription to a/b matches a publish to a/b → Your basic Trie insertion and lookup works.
  2. A subscription to a/+ matches a publish to a/b → Your single-level wildcard logic is correct.
  3. A subscription to a/# matches a/b, a/b/c, and a/b/c/d → Your multi-level wildcard logic is correct.
  4. The broker efficiently finds all matching subscribers without performance degradation → Your data structure and algorithm are sound.

Project 7: Client Library with Callbacks

  • File: LEARN_MQTT_DEEP_DIVE.md
  • Main Programming Language: C
  • Alternative Programming Languages: C++
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 2. The “Micro-SaaS / Pro Tool”
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Library Design / API / Function Pointers
  • Software or Tool: Your existing client code.
  • Main Book: “C Interfaces and Implementations” by David R. Hanson.

What you’ll build: Refactor your client-side code into a proper, reusable C library (e.g., libmymqtt.so). Instead of a monolithic main function, you will expose an API and use function pointers for callbacks to handle events like on_connect, on_message, and on_disconnect.

Why it teaches MQTT: This is how all professional MQTT client libraries (like Paho and Mosquitto) are designed. It teaches you to separate the low-level protocol and network logic from the application logic, creating a clean, event-driven API that is easy for other developers to use.

Core challenges you’ll face:

  • API Design → maps to defining a clean, intuitive set of functions for your library, e.g., mqtt_new(), mqtt_connect(), mqtt_publish(), mqtt_subscribe(), mqtt_loop().
  • Callback Mechanism using Function Pointers → maps to designing structs to hold user-provided callback functions and a void* for user data.
  • Encapsulation → maps to hiding internal state (like the socket descriptor and packet buffers) in an opaque struct, exposing it only through your API.
  • Creating a Shared Library → maps to compiling your code into a .so (Linux) or .dylib (macOS) file and writing a separate application that links against it.

Key Concepts:

  • Opaque Pointers (Information Hiding): A common C idiom where you expose struct MyMqttClient; in the header but define it only in the .c file.
  • Function Pointers: “The C Programming Language” by Kernighan & Ritchie, Chapter 5.
  • Shared Library Creation with GCC/Clang: Using the -shared and -fPIC compiler flags.

Difficulty: Advanced Time estimate: 2 weeks Prerequisites: Project 2.

Real world outcome: You will have a clean header file mymqtt.h and a small example program that uses your library.

mymqtt.h:

// Conceptual API
struct mqtt_client;
typedef struct mqtt_client mqtt_client_t;

// Define callback function types
typedef void (on_message_callback)(void* user_data, const char* topic, const char* payload);

mqtt_client_t* mqtt_new(const char* host, int port, const char* client_id);
void mqtt_set_on_message(mqtt_client_t* client, on_message_callback* cb, void* user_data);
int mqtt_connect(mqtt_client_t* client);
int mqtt_publish(mqtt_client_t* client, const char* topic, const char* message, int qos);
int mqtt_loop(mqtt_client_t* client, int timeout); // Processes network events
void mqtt_free(mqtt_client_t* client);

example.c:

#include "mymqtt.h"
#include <stdio.h>

void my_message_handler(void* user_data, const char* topic, const char* payload) {
    printf("Received message on topic '%s': %s\n", topic, payload);
}

int main() {
    mqtt_client_t* client = mqtt_new("test.mosquitto.org", 1883, "my-lib-client");
    mqtt_set_on_message(client, my_message_handler, NULL);
    mqtt_connect(client);
    mqtt_subscribe(client, "some/topic", 0);
    while (1) {
        mqtt_loop(client, 1000); // Loop forever, processing events
    }
    mqtt_free(client);
    return 0;
}

Learning milestones:

  1. Code is separated into a library and an application → You understand the principles of modular design.
  2. The application receives a message via a callback function → Your event-driven architecture works.
  3. The library hides all its internal state → You have created a clean, maintainable API.
  4. Another developer could easily use your library without understanding its internals → You have succeeded in building a high-quality, reusable component.

Project 8: Broker with Persistence (Retained Messages)

  • File: LEARN_MQTT_DEEP_DIVE.md
  • Main Programming Language: C
  • Alternative Programming Languages: Go, Rust
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 4. The “Open Core” Infrastructure
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Server Architecture / State Persistence
  • Software or Tool: Your broker.
  • Main Book: “The Linux Programming Interface” by Michael Kerrisk (for file I/O).

What you’ll build: Enhance your broker to support the “Retain” flag on PUBLISH packets. When a message is published with this flag, the broker must store it. The moment a new client subscribes to a matching topic, the broker must immediately send them the last retained message for that topic.

Why it teaches MQTT: Retained messages are a crucial feature for IoT state synchronization. A new device coming online needs to know the last known state of a system (e.g., “is the light on or off?”) without waiting for a new message. This project teaches you how to add a layer of persistence to your broker’s message handling.

Core challenges you’ll face:

  • Detecting the Retain Flag → maps to correctly parsing the first byte of the fixed header of a PUBLISH packet.
  • Storing Retained Messages → maps to designing a data structure (e.g., a hash map from topic to message content) to store the retained messages.
  • Handling Deletion of Retained Messages → maps to implementing the rule that publishing a zero-byte message with the retain flag on a topic clears the retained message for that topic.
  • Sending Retained Messages on Subscribing → maps to modifying your SUBSCRIBE logic to immediately check for and send any matching retained messages to the new subscriber.

Key Concepts:

  • Retained Messages: MQTT v3.1.1 Specification, Section 4.1.
  • File I/O for Persistence: To make retained messages survive a broker restart, you’ll need to write them to a file (e.g., in a simple text-based format or using a small embedded database like SQLite).
  • In-Memory vs. Disk Storage: Understanding the trade-offs between performance (in-memory) and durability (disk).

Difficulty: Advanced Time estimate: 2 weeks

  • Prerequisites: Project 4.

Real world outcome:

  1. Client A publishes a message {"status": "ON"} to home/light/1 with the Retain flag.
  2. Client A disconnects.
  3. Five minutes later, Client B connects and subscribes to home/light/1.
  4. Client B immediately receives the message {"status": "ON"} from the broker without Client A having to republish it.

Implementation Hints: You will need a new data structure in your broker:

// Conceptual storage
struct RetainedMessage {
    char* topic;
    char* payload;
    size_t payload_len;
};

// A hash map is a good choice for the main storage
// GHashTable from GLib or write your own.
// Key: topic string, Value: struct RetainedMessage*

Your logic will change in two places:

  1. In the PUBLISH handler: If the Retain flag is set, you will either add/update the entry in your retained message storage. If the payload length is zero, you will remove the entry.
  2. In the SUBSCRIBE handler: After adding the client to the list of subscribers for a topic filter, you must then iterate through your retained message storage. For every retained message whose topic matches the filter, you send a PUBLISH packet to the newly subscribed client.

For extra credit, make this persistent by serializing the retained messages to a file whenever they change, and loading this file when the broker starts.

Learning milestones:

  1. Broker correctly stores a retained message → Your storage mechanism works.
  2. A new subscriber immediately receives the correct retained message → Your SUBSCRIBE logic is correct.
  3. Publishing an empty retained message clears the old one → You’ve implemented the deletion rule.
  4. (Optional) Retained messages survive a broker restart → You have implemented true persistence.

Project 9: MQTT-to-WebSocket Bridge

  • File: LEARN_MQTT_DEEP_DIVE.md
  • Main Programming Language: Go or Python (easier for this task)
  • Alternative Programming Languages: C (much harder), Node.js
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 3. The “Service & Support” Model
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Protocol Interoperability / Web Technologies
  • Software or Tool: A WebSocket library (e.g., gorilla/websocket for Go) and your MQTT client library.
  • Main Book: “Design and Build Great Web APIs” by Mike Amundsen.

What you’ll build: A standalone application that acts as a bridge. It will connect to your MQTT broker and subscribe to a topic. Simultaneously, it will run a WebSocket server. When it receives a message from the MQTT broker, it will forward it to all connected WebSocket clients.

Why it teaches MQTT: This is an extremely common real-world use case. Backend IoT services run on MQTT, but web-based dashboards and frontends run on WebSockets. This project teaches you how to bridge these two worlds, showing how MQTT fits into a larger architecture.

Core challenges you’ll face:

  • Managing Two Network Protocols at Once → maps to using your MQTT client library to connect to the broker while also using a WebSocket library to accept browser connections.
  • Concurrency → maps to using goroutines (in Go) or threads/asyncio (in Python) to handle the MQTT client loop and the WebSocket server loop simultaneously.
  • Data Transformation → maps to deciding on a JSON format to wrap the MQTT message (e.g., {"topic": "...", "payload": "..."}) before sending it over the WebSocket.
  • Fan-out Logic → maps to when one MQTT message comes in, you need to iterate over all connected WebSocket clients and send it to each one.

Key Concepts:

  • WebSockets: A protocol providing full-duplex communication channels over a single TCP connection, commonly used by web browsers.
  • Goroutines and Channels (Go): A powerful and simple concurrency model perfect for this kind of I/O-bound task.
  • Protocol Gateway/Bridge Pattern: A common architectural pattern for translating between different communication protocols.

Difficulty: Advanced Time estimate: 1-2 weeks Prerequisites: Project 7 (Client Library), basic knowledge of WebSockets. Go is highly recommended as its concurrency model makes this project much simpler.

Real world outcome: You can build a simple HTML page with JavaScript that connects to your bridge. When a sensor publishes to your C-based MQTT broker, the value instantly appears on the web page.

Web Page (index.html):

<script>
    let ws = new WebSocket("ws://localhost:8080/ws");
    ws.onmessage = function(event) {
        let msg = JSON.parse(event.data);
        document.body.innerHTML += `New message on ${msg.topic}: ${msg.payload}<br>`;
    };
</script>

Bridge Application Logs:

$ ./mqtt_ws_bridge
MQTT client connected. Subscribed to 'sensors/#'.
WebSocket server listening on :8080.
MQTT message received on 'sensors/temp'. Forwarding to 3 WebSocket clients.

Implementation Hints (in Go):

// Conceptual Go implementation
func main() {
    // 1. Connect to MQTT broker (using a Go MQTT library)
    // 2. Subscribe to a topic. In the on-message callback,
    //    push the message into a Go channel.

    // 3. Set up a WebSocket server.
    //    - Keep a map of connected clients.
    //    - The HTTP handler upgrades the connection to a WebSocket.

    // 4. Start a goroutine that reads from the MQTT channel.
    go func() {
        for msg := range mqttChannel {
            // For each connected WebSocket client...
            //   ...send them the message.
        }
    }()

    // 5. Block forever, running the server.
}

This structure decouples the MQTT part from the WebSocket part, making the logic clean and easy to manage.

Learning milestones:

  1. The bridge connects to both MQTT and a WebSocket client → Both networking layers are functional.
  2. An MQTT message is successfully received by the bridge → The MQTT client logic is working.
  3. The message is successfully forwarded to a connected web browser → The end-to-end pipeline is complete.
  4. The bridge handles multiple WebSocket clients connecting and disconnecting without errors → Your concurrent state management is robust.

Project 10: Securing Communication with TLS

  • File: LEARN_MQTT_DEEP_DIVE.md
  • Main Programming Language: C
  • Alternative Programming Languages: Go, Rust
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 3. The “Service & Support” Model
  • Difficulty: Level 4: Expert
  • Knowledge Area: Network Security / Cryptography
  • Software or Tool: OpenSSL library (libssl, libcrypto).
  • Main Book: “Serious Cryptography” by Jean-Philippe Aumasson.

What you’ll build: Modify both your MQTT client and broker to communicate over TLS (Transport Layer Security). This involves wrapping your raw TCP sockets with an OpenSSL layer to encrypt all traffic, preventing eavesdropping and man-in-the-middle attacks.

Why it teaches MQTT: In any real-world deployment, MQTT traffic must be encrypted. This project teaches you the standard way to do it. You’ll learn the TLS handshake process and how to integrate a complex security library like OpenSSL into a C networking application.

Core challenges you’ll face:

  • Learning the OpenSSL API → maps to understanding the opaque and complex nature of OpenSSL, including objects like SSL_CTX and SSL.
  • TLS Handshake Implementation → maps to correctly performing the client and server-side of the TLS handshake before sending any MQTT data.
  • Certificate and Key Management → maps to generating self-signed certificates for testing and loading them into your applications.
  • Replacing send/recv with SSL_write/SSL_read → maps to adapting your existing network code to use OpenSSL’s I/O functions instead of the standard socket calls.

Key Concepts:

  • Transport Layer Security (TLS): The standard protocol for encrypting network communication, successor to SSL.
  • Public Key Infrastructure (PKI): The system of certificates, private keys, and certificate authorities that underpins TLS.
  • OpenSSL: The most widely used open-source cryptography and TLS library.

Difficulty: Expert Time estimate: 2-3 weeks Prerequisites: Project 4, a willingness to read dense API documentation.

Real world outcome: You can now run your broker and client on a secure port (typically 8883 for MQTT over TLS). If you inspect the traffic with Wireshark, all the MQTT packets will be unreadable, shown only as “Application Data” within TLS records.

Broker Startup:

$ ./my_secure_broker --cert server.crt --key server.key
Secure MQTT Broker started on port 8883.

Client Connection:

$ ./my_secure_client --host localhost --port 8883 --cafile ca.crt
Secure connection established. Proceeding with MQTT handshake.

Implementation Hints:

The general workflow with OpenSSL is:

Server Side:

  1. Initialize OpenSSL library.
  2. Create an SSL_CTX (SSL Context) object.
  3. Load your server certificate and private key into the context using SSL_CTX_use_certificate_file and SSL_CTX_use_PrivateKey_file.
  4. After you accept() a new client TCP socket…
  5. Create a new SSL object from the context using SSL_new().
  6. Attach the TCP socket to the SSL object with SSL_set_fd().
  7. Perform the TLS handshake with SSL_accept().
  8. Use SSL_read() and SSL_write() instead of recv() and send().
  9. When done, shut down with SSL_shutdown() and SSL_free().

Client Side: The process is similar, but you use SSL_connect() instead of SSL_accept() and you load the CA certificate to verify the server’s identity.

A key difficulty is OpenSSL’s error handling. It uses its own error queue, which you must check with functions like ERR_get_error() and ERR_error_string().

Learning milestones:

  1. The TLS handshake completes successfully → You’ve correctly set up the certificates and OpenSSL contexts.
  2. An encrypted MQTT CONNECT packet is sent and acknowledged → You have successfully layered the MQTT protocol on top of a TLS stream.
  3. Your previously plain-text communication is now fully encrypted → You can verify this with Wireshark.
  4. The system handles TLS-level errors gracefully → Your error handling is robust enough for production-like environments.

Summary

| Project | Main Language | | :— | :— | | 1. MQTT Packet Parser | C | | 2. Simple CONNECT Client | C | | 3. Command-Line PUBLISH and SUBSCRIBE Client | C | | 4. Concurrent In-Memory Broker | C | | 5. Implementing QoS 1 | C | | 6. Broker with Topic Wildcard Support | C | | 7. Client Library with Callbacks | C | | 8. Broker with Persistence (Retained Messages) | C | | 9. MQTT-to-WebSocket Bridge | Go or Python | | 10. Securing Communication with TLS | C |

```