← Back to all projects

LEARN UNICODE DEEP DIVE

Learn Unicode: From ASCII to Emojis

Goal: To deeply understand how modern text is represented in computers. You’ll master the difference between characters and bytes, decode UTF-8 by hand, and build tools that handle text correctly.

Why Learn Unicode?

Nearly every developer has been bitten by an encoding bug: garbled text (mojibake), incorrect string lengths, or broken special characters. Most of the time, we treat text as a magical black box. This learning journey is about prying that box open.

Understanding Unicode and UTF-8 isn’t just about fixing bugs; it’s about understanding a fundamental abstraction of modern computing. After completing these projects, you will:

  • Never be confused by “character,” “byte,” “code point,” and “grapheme” again.
  • Be able to look at a sequence of hex bytes and know if it’s valid UTF-8.
  • Understand why string.length() can be so misleading.
  • Build tools that manipulate text correctly, regardless of the language or characters.
  • Grasp the challenges of representing all human language in a digital format.

Core Concept Analysis

The Four-Layer Model of Text

  1. Layer 1: The Abstract Character (Grapheme Cluster)
    • What a user thinks of as a single character.
    • Examples: A, é, 👍, 👨‍👩‍👧‍👦 (a single family emoji).
    • This is the hardest to measure and is context-dependent.
  2. Layer 2: The Code Point
    • A unique number assigned by the Unicode Consortium to a conceptual character or symbol. This is the “atomic unit” of Unicode.
    • Written in the format U+<hex_number>.
    • Examples: A is U+0041. é is U+00E9. 👍 is U+1F44D.
    • A single Grapheme Cluster can be composed of multiple Code Points (e.g., e + ´ combining accent). The family emoji is composed of 7 code points!
  3. Layer 3: The Character Encoding (UTF-8, UTF-16)
    • A rule for how to represent a code point’s number as a sequence of bytes. This is where the magic happens.
    • UTF-8: A variable-width encoding. Uses 1-4 bytes. ASCII characters are 1 byte. More common characters are 2-3 bytes. Emojis are often 4 bytes. This is the dominant encoding of the web.
    • UTF-16: A variable-width encoding. Uses 2 or 4 bytes. Used internally by Windows, Java, and JavaScript.
    • UTF-32: A fixed-width encoding. Every code point is 4 bytes. Simple, but wastes space.
  4. Layer 4: The Bytes
    • The raw 8-bit numbers that are actually stored on disk or sent over the network.
    • Example: The code point for € (U+20AC) is represented in UTF-8 as the three bytes E2 82 AC.

UTF-8 Encoding Rules

This is the most important encoding to understand.

Code Point Range UTF-8 Byte Sequence (Binary)
U+0000 to U+007F 0xxxxxxx (1 byte, identical to ASCII)
U+0080 to U+07FF 110xxxxx 10xxxxxx (2 bytes)
U+0800 to U+FFFF 1110xxxx 10xxxxxx 10xxxxxx (3 bytes)
U+10000 to U+10FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx (4 bytes)

The x’s are where the bits of the code point number are stored. The leading bits of the first byte tell you how many bytes are in the sequence. Subsequent bytes all start with 10. This makes the encoding self-synchronizing and easy to validate.

Project List

These projects are designed to be done in a low-level language like C or Rust to force you to confront the byte-level realities of text. C is recommended for the full manual experience.

Project 1: An ASCII-only Hex Dump Tool

  • File: LEARN_UNICODE_DEEP_DIVE.md
  • Main Programming Language: C
  • Alternative Programming Languages: Python, Go
  • Coolness Level: Level 2: Practical but Forgettable
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 1: Beginner
  • Knowledge Area: Binary Data / File I/O
  • Software or Tool: A hex editor for verification
  • Main Book: “The C Programming Language” by Kernighan & Ritchie (K&R)

What you’ll build: A command-line tool that reads any file and prints its contents as a “hex dump”: the byte offset, the hexadecimal representation of the bytes, and their printable ASCII equivalents.

Why it teaches Unicode: It establishes the baseline and exposes the fundamental flaw in the “1 byte = 1 character” assumption. The tool will work perfectly for ascii.txt but will produce garbled nonsense in the character column for utf8.txt. This failure is the entire point of the lesson.

Core challenges you’ll face:

  • Reading a file byte-by-byte → maps to using fread or fgetc in a loop
  • Formatting output in columns → maps to using printf with specific width and padding
  • Distinguishing printable vs. non-printable ASCII → maps to checking if a byte’s value is between 32 and 126

Key Concepts:

  • File I/O: Chapter 7 of K&R.
  • ASCII Table: You’ll need to know which byte values correspond to which characters.
  • Integer Formatting: printf format specifiers like %02X for hex.

Difficulty: Beginner Time estimate: A few hours Prerequisites: Basic C programming.

Real world outcome: A tool that produces output like this:

$ ./hexdump file.txt
00000000: 48 65 6c 6c 6f 20 57 6f 72 6c 64 21 0a       Hello World!.
0000000D: e2 82 ac                                     ...

Notice how the Euro symbol € (bytes E2 82 AC) is rendered as three dots because those bytes are not printable ASCII characters.

Implementation Hints:

  1. Open the file in binary read mode: fopen(filename, "rb").
  2. Read the file in chunks (e.g., 16 bytes at a time) into a buffer.
  3. Loop through the buffer. For each byte, print its hex value.
  4. After printing the 16 hex values for the chunk, loop through the buffer again. This time, for each byte, check if it’s a printable ASCII character. If yes, print it. If no, print a dot (.).
  5. Keep track of the file offset and print it at the beginning of each line.

Learning milestones:

  1. The hex output matches a professional hex editor → You are correctly reading the raw bytes of a file.
  2. ASCII text is rendered correctly → You understand the 1-to-1 mapping in ASCII.
  3. UTF-8 text is rendered as gibberish → You have seen firsthand that the “1 byte = 1 character” model is broken.

Project 2: A UTF-8 Decoder

  • File: LEARN_UNICODE_DEEP_DIVE.md
  • Main Programming Language: C
  • Alternative Programming Languages: Rust
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Character Encoding / Bit Manipulation
  • Software or Tool: Your hex dump tool, a Unicode character table website
  • Main Book: “Unicode Explained” by Jukka K. Korpela

What you’ll build: A tool that reads a UTF-8 encoded file byte-by-byte and prints the corresponding Unicode code points.

Why it teaches Unicode: This is the core of the entire journey. It forces you to implement the UTF-8 decoding logic yourself. You’ll have to look at the leading bits of each byte to determine if it’s a single-byte character or the start of a multi-byte sequence, then assemble the final code point value.

Core challenges you’ll face:

  • Bitwise operations → maps to using & (AND), | (OR), and << (left shift) to inspect and assemble byte values
  • Implementing a state machine → maps to tracking “I’m in the middle of a 3-byte sequence, I need 2 more bytes”
  • Validating byte sequences → maps to checking for invalid sequences, like a continuation byte 10xxxxxx appearing at the start
  • Calculating the final code point value → maps to masking off the data bits from each byte and combining them into a single integer

Key Concepts:

  • Bit Manipulation: Any C tutorial on bitwise operators.
  • UTF-8 Encoding Rules: The table in the Core Concepts section above.
  • State Machines: Your code will be in one of a few states: “looking for a new character” or “looking for the Nth byte of a character”.

Difficulty: Advanced Time estimate: 1-2 days Prerequisites: Project 1, comfort with binary and hexadecimal.

Real world outcome: A tool that can process a file containing Hello € and produce this output:

$ ./utf8_decoder file.txt
Read byte: 48 -> Decoded U+0048 'H'
Read byte: 65 -> Decoded U+0065 'e'
Read byte: 6C -> Decoded U+006C 'l'
Read byte: 6C -> Decoded U+006C 'l'
Read byte: 6F -> Decoded U+006F 'o'
Read byte: 20 -> Decoded U+0020 ' '
Read 3 bytes: E2 82 AC -> Decoded U+20AC '€'

Implementation Hints:

  1. Read one byte at a time.
  2. Check the most significant bits to determine the byte’s role:
    • If (byte & 0x80) == 0, it’s a single-byte ASCII character. The code point is the byte itself.
    • If (byte & 0xE0) == 0xC0, it’s the start of a 2-byte sequence.
    • If (byte & 0xF0) == 0xE0, it’s the start of a 3-byte sequence.
    • If (byte & 0xF8) == 0xF0, it’s the start of a 4-byte sequence.
    • If (byte & 0xC0) == 0x80, it’s a continuation byte. This is an error if you weren’t expecting one.
  3. If you find a start byte, you know how many continuation bytes to read.
  4. For a multi-byte sequence, extract the data bits from each byte:
    • First byte: Mask off the leading control bits (e.g., byte & 0x1F for a 2-byte sequence).
    • Continuation bytes: Mask off the leading 10 (e.g., byte & 0x3F).
  5. Use bit-shifting to assemble the final code point. For a 3-byte sequence B1 B2 B3: codepoint = ((B1 & 0x0F) << 12) | ((B2 & 0x3F) << 6) | (B3 & 0x3F)

Learning milestones:

  1. ASCII text is decoded correctly → Your 1-byte logic is sound.
  2. 2 and 3-byte characters are decoded → You have mastered the core UTF-8 bit manipulation.
  3. 4-byte characters (like emoji) are decoded → Your logic scales to the full range of Unicode.
  4. The tool reports errors on invalid sequences → You have built a robust, validating decoder.

Project 3: A UTF-8 Encoder

  • File: LEARN_UNICODE_DEEP_DIVE.md
  • Main Programming Language: C
  • Alternative Programming Languages: Rust
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 3: Advanced
  • Knowledge Area: Character Encoding / Bit Manipulation
  • Software or Tool: Your decoder from Project 2.

What you’ll build: The reverse of the previous project. A tool that reads a list of code points (e.g., U+20AC U+1F600) and writes the raw UTF-8 bytes to a file.

Why it teaches Unicode: It solidifies your understanding of the encoding rules by forcing you to implement them in the other direction. It’s one thing to parse bytes; it’s another to generate them correctly.

Core challenges you’ll face:

  • Determining the number of bytes needed → maps to checking which range a code point falls into
  • Distributing code point bits into bytes → maps to more bit shifting and masking
  • Setting the UTF-8 control bits → maps to OR-ing your data bits with the correct prefixes like 0xC0, 0xE0, 0x80

Key Concepts:

  • UTF-8 Encoding Rules: The same table as before, but used for generation.
  • Integer Division and Modulo: Can be useful for extracting bits.
  • Binary Output: Writing raw byte values to a file, not their character representations.

Difficulty: Advanced Time estimate: 1 day Prerequisites: Project 2.

Real world outcome: A tool that can be used like this:

$ ./utf8_encoder "U+48 U+65 U+6c U+6c U+6f U+20ac" output.bin
$ hexdump output.bin
00000000: 48 65 6c 6c 6f e2 82 ac                   Hello...

You can then use your decoder from Project 2 to verify your encoder’s output.

Implementation Hints:

  1. Parse the input string to get a list of integer code points.
  2. For each code point:
    • If codepoint <= 0x7F, write 1 byte (the code point itself).
    • If codepoint <= 0x7FF, you need 2 bytes.
      • Byte 1: 0xC0 | (codepoint >> 6)
      • Byte 2: 0x80 | (codepoint & 0x3F)
    • If codepoint <= 0xFFFF, you need 3 bytes.
      • Byte 1: 0xE0 | (codepoint >> 12)
      • Byte 2: 0x80 | ((codepoint >> 6) & 0x3F)
      • Byte 3: 0x80 | (codepoint & 0x3F)
    • And so on for 4 bytes.
  3. Write these bytes to the output file opened in binary mode ("wb").

Learning milestones:

  1. Single-byte characters are encoded correctly.
  2. Multi-byte characters are encoded correctly.
  3. Your encoder’s output can be successfully decoded by your decoder. → You have a complete, working round-trip implementation of UTF-8.

Project 4: A “Character” Counter

  • File: LEARN_UNICODE_DEEP_DIVE.md
  • Main Programming Language: C or Rust
  • Alternative Programming Languages: Go, Python
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 2: Intermediate (if using a library for graphemes)
  • Knowledge Area: String Analysis / Unicode Standards
  • Software or Tool: A good Unicode library (like libunistring for C, or the standard libraries in Rust/Go)
  • Main Book: “Unicode Standard Annex #29: Unicode Text Segmentation”

What you’ll build: A tool that reads a text file and reports four different counts: (1) number of bytes, (2) number of ASCII characters, (3) number of Unicode code points (using your Project 2 logic), and (4) number of “perceived characters” (grapheme clusters).

Why it teaches Unicode: This project directly confronts the ambiguity of the word “character”. It proves, with data, that bytes, code points, and graphemes are different things. It’s the perfect demonstration for why string.length() is so often “wrong”.

Core challenges you’ll face:

  • Counting bytes → maps to simply getting the file size
  • Counting code points → maps to reusing your UTF-8 decoder logic
  • Counting grapheme clusters → maps to finding and using a library that correctly implements the Unicode segmentation algorithm
  • Finding interesting test cases → maps to finding emoji, combining characters, and multi-code-point flags to test your tool

Key Concepts:

  • Grapheme Cluster: The formal definition of a “user-perceived character”.
  • Unicode Segmentation: The official Unicode algorithm for determining word, line, and grapheme boundaries.
  • Combining Characters: Characters like accents that modify the preceding character (e.g., U+0301).

Difficulty: Intermediate Time estimate: 1 day Prerequisites: Project 2.

Real world outcome: Given a file containing just the Welsh flag emoji 🏴󠁧󠁢󠁷󠁬󠁳󠁿 (which is made of 7 code points), your tool would output:

$ ./char_counter flag.txt
Bytes: 28
ASCII Chars: 0
Code Points: 7
Grapheme Clusters (Perceived Chars): 1

Implementation Hints:

  1. Counting bytes is fseek and ftell or stat.
  2. Counting code points is the main loop from your Project 2.
  3. For grapheme clusters, you should not try to implement this yourself. The rules are incredibly complex.
    • In C, you might use a library like libunistring.
    • In Rust, the unicode-segmentation crate provides a graphemes(true) iterator.
    • In Swift, string.count gives you the grapheme count directly.
    • The learning here is using the library and seeing that the number is different from the code point count.
  4. Create test files with strings like: cafe vs café (the e and ´ are separate code points in the second version). Test with emoji like 👍🏽 (thumbs up + skin tone modifier).

Learning milestones:

  1. Byte and code point counts are correct → You have your own logic working.
  2. Grapheme count is different from code point count for test cases → You have proven the core concept.
  3. You can explain to someone else why a family emoji has a length of 7 in JavaScript but is one character.

Project 5: A “Safe” Substring Tool

  • File: LEARN_UNICODE_DEEP_DIVE.md
  • Main Programming Language: C or Rust
  • Alternative Programming Languages: Go
  • Coolness Level: Level 3: Genuinely Clever
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 3: Advanced
  • Knowledge Area: String Manipulation / UTF-8
  • Software or Tool: Your Project 2 decoder logic.

What you’ll build: A command-line tool that safely slices a UTF-8 string. It should take a start character index and an end character index (not byte indices) and print the corresponding substring.

Why it teaches Unicode: It forces you to solve a classic, thorny problem: you cannot simply slice a byte array without the risk of cutting a multi-byte character in half. This project teaches you to think of strings in terms of character boundaries, not byte offsets.

Core challenges you’ll face:

  • Iterating by character, not byte → maps to using your decoder logic to jump from the start of one character to the start of the next
  • Finding the byte offset for a character index → maps to looping N times through your decoder to find the byte position of the Nth character
  • Copying the correct range of bytes → maps to once you have the start and end byte offsets, using memcpy to extract the substring

Key Concepts:

  • Character Boundaries: The byte positions where a new UTF-8 sequence can start.
  • Iteration vs. Indexing: You can iterate through UTF-8 strings efficiently, but you cannot perform O(1) random access (indexing).
  • String Slicing: The correct way to implement substring logic.

Difficulty: Advanced Time estimate: 1 day Prerequisites: Project 2.

Real world outcome:

# The string is "你好世界" (Nǐ hǎo shìjiè)
$ ./safe_slice the_string.txt 1 3
好世

# A naive byte-based slice might have produced garbled output
# by cutting a 3-byte character incorrectly.

Implementation Hints:

  1. Read the entire file/string into a memory buffer.
  2. Implement a helper function, find_byte_offset_for_char_index(buffer, char_index).
  3. Inside this helper, loop from the beginning of the buffer. Use your UTF-8 decoder logic to determine the length of each character (1, 2, 3, or 4 bytes). Advance your byte pointer by that much in each iteration. Decrement a counter. When the counter hits zero, you’ve found the byte offset for the target character index.
  4. Call your helper function to get the start_byte_offset for the start_char_index and the end_byte_offset for the end_char_index.
  5. Calculate the length of the slice in bytes: byte_length = end_byte_offset - start_byte_offset.
  6. Use memcpy to copy that many bytes from buffer + start_byte_offset into a new buffer.
  7. Print the new buffer as the result.

Learning milestones:

  1. Slicing ASCII text works → Your baseline is correct.
  2. Slicing a string containing multi-byte characters works → You have successfully implemented character-based iteration.
  3. Your tool never produces invalid UTF-8 output → Your boundary logic is sound.

Project 6: A UTF-16 Decoder

  • File: LEARN_UNICODE_DEEP_DIVE.md
  • Main Programming Language: C
  • Alternative Programming Languages: Rust, Go
  • Coolness Level: Level 4: Hardcore Tech Flex
  • Business Potential: 1. The “Resume Gold”
  • Difficulty: Level 4: Expert
  • Knowledge Area: Character Encoding / Endianness
  • Software or Tool: A hex editor.
  • Main Book: “Unicode Explained” by Jukka K. Korpela

What you’ll build: A tool that decodes a UTF-16 encoded file. It must correctly handle endianness (via the BOM) and surrogate pairs.

Why it teaches Unicode: It expands your knowledge beyond the world of UTF-8. You’ll learn about the Byte Order Mark (BOM), endianness, and the complex “surrogate pair” mechanism that UTF-16 uses to represent code points above U+FFFF. This is essential for understanding text handling on Windows or in Java/JavaScript.

Core challenges you’ll face:

  • Detecting the BOM → maps to reading the first two bytes (FF FE for little-endian, FE FF for big-endian) to determine byte order
  • Handling Endianness → maps to swapping bytes if necessary when reading each 16-bit code unit
  • Decoding Surrogate Pairs → maps to the most complex part: identifying a “high surrogate” (0xD800-0xDBFF), reading the following “low surrogate” (0xDC00-0xDFFF), and performing the mathematical calculation to get the final code point

Key Concepts:

  • Byte Order Mark (BOM): A special sequence to indicate encoding and endianness.
  • Endianness: The order in which bytes are arranged to form a larger integer.
  • Surrogate Pairs: The “hack” used by UTF-16 to access code points beyond the Basic Multilingual Plane.

Difficulty: Expert Time estimate: 2 days Prerequisites: Project 2, strong understanding of bit manipulation.

Real world outcome: Given a UTF-16 file containing a single emoji 😂 (U+1F602), which is encoded in UTF-16 (LE) as the bytes 3D D8 02 DE, your tool would output:

$ ./utf16_decoder emoji_utf16.txt
BOM: Little Endian
Read surrogate pair: D83D DE02 -> Decoded U+1F602 '😂'

Implementation Hints:

  1. Read the first two bytes. If they match a BOM, set your endianness mode. If not, you may have to guess (little-endian is most common).
  2. Read the file in 2-byte (16-bit) chunks. If your endianness mode requires it, swap the bytes.
  3. For each 16-bit code unit:
    • If it’s in the range 0xD800 to 0xDBFF, it’s a high surrogate. You must immediately read the next 16-bit unit, which must be a low surrogate (0xDC00 to 0xDFFF).
    • If it’s a surrogate pair, use the formula to calculate the code point: codepoint = 0x10000 + ((high - 0xD800) << 10) | (low - 0xDC00)
    • If it’s not a surrogate, it’s a code point from the BMP, and its value is simply the 16-bit unit itself.

Learning milestones:

  1. The BOM is correctly detected.
  2. BMP characters from a UTF-16 file are decoded correctly.
  3. Emoji and other characters above U+FFFF are decoded correctly → You have mastered surrogate pairs.

Summary

Project Difficulty Time Main Concept Taught
1. ASCII Hex Dump Beginner Hours The “1 byte = 1 char” fallacy
2. UTF-8 Decoder Advanced 1-2 Days UTF-8 bit-level mechanics
3. UTF-8 Encoder Advanced 1 Day Generating valid UTF-8 sequences
4. “Character” Counter Intermediate 1 Day Bytes vs. Code Points vs. Graphemes
5. “Safe” Substring Advanced 1 Day Character-aware string slicing
6. UTF-16 Decoder Expert 2 Days Endianness and Surrogate Pairs