Project 1: musl Compatibility Lab
Quick Reference
| Attribute | Details |
|---|---|
| Difficulty | Intermediate |
| Time Estimate | 1 week (20-30 hours) |
| Primary Language | C |
| Alternative Languages | Go, Rust, Python (with C extensions) |
| Knowledge Area | C Library / Binary Compatibility |
| Software/Tools | Alpine Linux, Docker, GCC, gdb |
| Main Book | “The Linux Programming Interface” by Michael Kerrisk |
| Prerequisites | Basic C programming, Docker fundamentals |
Learning Objectives
By completing this project, you will:
- Understand what a C library is and why every compiled program depends on one
- Diagnose binary compatibility issues between glibc and musl systems
- Master thread stack management and understand musl’s 128 KB default vs glibc’s 2-8 MB
- Identify DNS resolution differences between musl and glibc resolvers
- Handle locale and Unicode differences when code relies on locale-specific behavior
- Apply multiple strategies for running glibc binaries on Alpine (static linking, gcompat, recompilation)
- Write portable C code that works correctly on both musl and glibc systems
Theoretical Foundation
Core Concepts
1. What Is a C Library?
Every compiled program—whether written in C, C++, Go, Rust, or any language that compiles to native code—needs to communicate with the operating system. The C library provides this interface:
┌─────────────────────────────────────────────────────────────────────┐
│ C Library Architecture │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Your Program (C, C++, Go, Rust, etc.) │
│ │ │
│ │ calls: malloc(), printf(), pthread_create(), │
│ │ fopen(), socket(), getaddrinfo()... │
│ ▼ │
│ ┌─────────────────────────────────────────────────┐ │
│ │ C Library (libc) │ │
│ │ │ │
│ │ musl (Alpine) glibc (Ubuntu/Debian) │ │
│ │ ───────────── ───────────────────── │ │
│ │ ~1 MB ~8 MB │ │
│ │ Minimal, strict POSIX Feature-rich, GNU ext │ │
│ │ 128 KB thread stack 2-8 MB thread stack │ │
│ │ Simple DNS resolver Complex caching DNS │ │
│ │ │ │
│ └─────────────────────────────────────────────────┘ │
│ │ │
│ │ system calls: read(), write(), mmap(), │
│ │ clone(), futex()... │
│ ▼ │
│ ┌─────────────────────────────────────────────────┐ │
│ │ Linux Kernel │ │
│ └─────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
Key Functions Provided by libc:
| Category | Functions | Purpose |
|---|---|---|
| Memory | malloc(), free(), realloc(), mmap() |
Dynamic memory allocation |
| I/O | printf(), fopen(), read(), write() |
File and console I/O |
| Strings | strlen(), strcpy(), strcmp() |
String manipulation |
| Threading | pthread_create(), pthread_mutex_* |
POSIX threads |
| Networking | socket(), connect(), getaddrinfo() |
Network communication |
| DNS | gethostbyname(), getaddrinfo() |
Name resolution |
| Time | time(), gettimeofday(), nanosleep() |
Time operations |
| Process | fork(), exec(), waitpid() |
Process management |
2. Dynamic vs Static Linking
When you compile a program, the linker must resolve references to libc functions:
┌─────────────────────────────────────────────────────────────────────┐
│ Dynamic vs Static Linking │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ DYNAMIC LINKING (default) │
│ ───────────────────────── │
│ │
│ Binary (small) At Runtime │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Your Code │ │ Your Code │ │
│ │ + References │ ────────►│ + libc.so │ │
│ │ to libc │ load │ (from disk) │ │
│ └──────────────┘ └──────────────┘ │
│ │
│ Pros: Small binaries, shared memory, easy updates │
│ Cons: Requires correct libc version at runtime │
│ │
│ ─────────────────────────────────────────────────────── │
│ │
│ STATIC LINKING │
│ ────────────── │
│ │
│ Binary (large) │
│ ┌──────────────┐ │
│ │ Your Code │ │
│ │ + ALL libc │ ←── Everything included, no runtime deps │
│ │ functions │ │
│ └──────────────┘ │
│ │
│ Pros: Runs anywhere, no dependencies │
│ Cons: Larger binaries, no shared memory, harder to update │
│ │
└─────────────────────────────────────────────────────────────────────┘
Why This Matters for Alpine:
A dynamically-linked glibc binary looks for libc.so.6 at a specific path (/lib/x86_64-linux-gnu/libc.so.6). Alpine has musl at /lib/ld-musl-x86_64.so.1. The binary simply cannot find its required library.
3. Thread Stack Size Differences
This is one of the most impactful practical differences:
┌─────────────────────────────────────────────────────────────────────┐
│ Thread Stack Size Comparison │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ glibc Default: 2-8 MB per thread │
│ ┌────────────────────────────────────────────────────────┐ │
│ │████████████████████████████████████████████████████████│ 8 MB │
│ └────────────────────────────────────────────────────────┘ │
│ │
│ musl Default: 128 KB per thread │
│ ┌────┐ │
│ │████│ 128 KB │
│ └────┘ │
│ │
│ Stack Usage Example: │
│ ─────────────────── │
│ char local_buffer[1024 * 1024]; // 1 MB local variable │
│ │
│ glibc: ✓ Fits easily within 8 MB stack │
│ musl: ✗ CRASH! Stack overflow (1 MB > 128 KB) │
│ │
│ Common Symptoms: │
│ - SIGSEGV (Segmentation fault) │
│ - "Stack smashing detected" │
│ - Random crashes in threaded code │
│ - Works on Ubuntu, crashes on Alpine │
│ │
└─────────────────────────────────────────────────────────────────────┘
Why musl Uses 128 KB:
musl was designed for embedded systems where:
- Memory is precious
- 100+ threads might run simultaneously
- 8 MB × 100 threads = 800 MB just for stacks!
- 128 KB × 100 threads = 12.5 MB (much more reasonable)
4. DNS Resolver Differences
The DNS resolver is another major difference that causes real-world issues:
┌─────────────────────────────────────────────────────────────────────┐
│ DNS Resolver Architecture │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ glibc Resolver (nscd, glibc-internal) │
│ ───────────────────────────────────── │
│ ┌──────────────────────────────────────────────┐ │
│ │ │ │
│ │ • Parallel queries to multiple DNS servers│ │
│ │ • Response caching │ │
│ │ • Complex search domain handling │ │
│ │ • IPv4/IPv6 "happy eyeballs" algorithm │ │
│ │ • Sophisticated timeout/retry logic │ │
│ │ • NSS (Name Service Switch) support │ │
│ │ │ │
│ └──────────────────────────────────────────────┘ │
│ │
│ musl Resolver │
│ ───────────── │
│ ┌──────────────────────────────────────────────┐ │
│ │ │ │
│ │ • Single-threaded, sequential queries │ │
│ │ • No caching (relies on system cache) │ │
│ │ • Simple search domain handling │ │
│ │ • Basic timeout behavior │ │
│ │ • Strict standards compliance │ │
│ │ │ │
│ └──────────────────────────────────────────────┘ │
│ │
│ Real-World Impact: │
│ ───────────────── │
│ • High-concurrency DNS lookups may be slower on musl │
│ • Search domains in /etc/resolv.conf behave differently │
│ • IPv4/IPv6 ordering may differ │
│ • Timeout behavior under network issues varies │
│ │
└─────────────────────────────────────────────────────────────────────┘
5. Locale and Character Handling
musl takes a minimalist approach to locales:
┌─────────────────────────────────────────────────────────────────────┐
│ Locale Support Comparison │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ glibc Locales │
│ ───────────── │
│ $ locale -a │
│ C │
│ C.UTF-8 │
│ en_US.UTF-8 │
│ fr_FR.UTF-8 │
│ de_DE.UTF-8 │
│ ja_JP.UTF-8 │
│ ... (100+ locales) │
│ │
│ Features: │
│ • Locale-specific string sorting (collation) │
│ • Locale-specific number formatting (1,000.00 vs 1.000,00) │
│ • Locale-specific date formatting │
│ • Character classification per locale │
│ │
│ ─────────────────────────────────────────────────────── │
│ │
│ musl Locales │
│ ──────────── │
│ $ locale -a │
│ C │
│ C.UTF-8 │
│ POSIX │
│ │
│ Features: │
│ • UTF-8 focused (handles Unicode correctly) │
│ • Minimal locale-specific behavior │
│ • Consistent behavior regardless of LANG setting │
│ │
│ Impact on Code: │
│ ───────────────── │
│ • strcoll() behaves as strcmp() (no locale-aware sorting) │
│ • setlocale() succeeds but has limited effect │
│ • Character classification works for UTF-8 │
│ │
└─────────────────────────────────────────────────────────────────────┘
Why This Matters
The Real-World Problem:
You develop an application on Ubuntu. It works perfectly:
- Handles high-traffic DNS lookups efficiently
- Spawns 50 worker threads without issue
- Sorts strings according to user locale
- Runs binaries from third-party vendors
You deploy to Alpine for its small container size. Suddenly:
- DNS lookups timeout under load
- Random crashes in worker threads (stack overflow)
- String sorting produces unexpected results
- Third-party binaries fail with “not found” errors
Industry Impact:
According to Docker Hub statistics:
- Alpine is one of the most popular base images
- Many production issues stem from glibc/musl differences
- Understanding these differences saves hours of debugging
Historical Context
glibc (GNU C Library):
- First released: 1988 (as GNU libc)
- Current maintainer: GNU Project
- Design philosophy: Feature-rich, backwards compatible
- Target: Desktop and server Linux
musl:
- First released: 2011 by Rich Felker
- Design philosophy: Correct, minimal, efficient
- Target: Embedded systems, containers, security-focused systems
- Name origin: “musl” is pronounced “muscle”
Why Alpine Chose musl:
- Size: Alpine’s goal is minimal images
- Security: Smaller codebase = smaller attack surface
- Simplicity: Easier to audit, fewer historic CVEs
- Standards: Strict POSIX compliance catches bugs
Common Misconceptions
Misconception 1: “musl is just a smaller glibc”
Reality: musl is a completely different implementation of the C library. It’s not a stripped-down glibc—it’s written from scratch with different design decisions.
Misconception 2: “Static linking solves all problems”
Reality: Static linking helps with binary compatibility but:
- Some libraries (like glibc’s NSS) don’t work statically
- Go programs with CGO still need the C library
- Python with native extensions needs compatible libraries
Misconception 3: “Just install gcompat and everything works”
Reality: gcompat provides a compatibility layer, but:
- It doesn’t emulate all glibc behavior
- Performance may differ
- Some edge cases still fail
- Better to compile for musl when possible
Misconception 4: “musl is slower than glibc”
Reality: It depends on the workload:
- musl malloc is often faster for small allocations
- glibc can be faster for specific optimized operations
- DNS resolution differs in design, not necessarily speed
Project Specification
What You Will Build
A comprehensive test suite that demonstrates and measures the key differences between musl and glibc. The suite includes:
- Binary Compatibility Tester - Demonstrates why glibc binaries fail on Alpine
- Thread Stack Tester - Shows stack size differences and crashes
- DNS Resolution Benchmark - Compares resolver behavior under load
- Locale Behavior Tester - Shows locale handling differences
- Compatibility Fix Demonstrator - Shows solutions for each issue
Functional Requirements
- Binary Compatibility Tests:
- Compile a simple binary on Ubuntu/glibc
- Attempt to run on Alpine/musl (demonstrate failure)
- Show the exact error messages and explain them
- Demonstrate solutions: static linking, gcompat, recompilation
- Thread Stack Tests:
- Create threads with varying local variable sizes
- Demonstrate stack overflow on musl with default settings
- Show how to fix with
pthread_attr_setstacksize() - Measure actual stack usage
- DNS Resolution Tests:
- Benchmark concurrent DNS lookups
- Compare timing between musl and glibc
- Test search domain behavior
- Test failure/timeout scenarios
- Locale Tests:
- Test string sorting with different locales
- Test character classification
- Test number/date formatting
Non-Functional Requirements
- All tests must be reproducible via Docker
- Tests should include clear pass/fail criteria
- Output should be human-readable with explanations
- Code should be well-commented for learning
Example Usage / Output
# Run the complete test suite
$ ./run-tests.sh
╔══════════════════════════════════════════════════════════════╗
║ musl vs glibc Compatibility Lab ║
╚══════════════════════════════════════════════════════════════╝
┌──────────────────────────────────────────────────────────────┐
│ Test 1: Binary Compatibility │
├──────────────────────────────────────────────────────────────┤
Compiling hello.c on Ubuntu (glibc)...
$ gcc -o hello-glibc hello.c
✓ Compilation successful
Running on Ubuntu:
$ ./hello-glibc
Hello from glibc-compiled binary!
✓ Runs successfully
Copying to Alpine and running:
$ docker cp hello-glibc alpine:/tmp/
$ docker exec alpine /tmp/hello-glibc
/tmp/hello-glibc: error while loading shared libraries:
libc.so.6: cannot open shared object file: No such file or directory
✗ EXPECTED FAILURE: Binary cannot find glibc
Inspecting the binary:
$ ldd hello-glibc
linux-vdso.so.1 (0x00007ffd...)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f...)
/lib64/ld-linux-x86-64.so.2 (0x00007f...)
Explanation:
The binary was dynamically linked against glibc.
It expects to find libc.so.6 at /lib/x86_64-linux-gnu/
Alpine has musl at /lib/ld-musl-x86_64.so.1 instead.
Solutions demonstrated:
1. Static linking: CGO_ENABLED=0 go build
2. Recompile on Alpine: apk add build-base && gcc -o hello hello.c
3. Compatibility layer: apk add gcompat
└──────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────┐
│ Test 2: Thread Stack Size │
├──────────────────────────────────────────────────────────────┤
Test: 256 KB local variable in thread
On Ubuntu (glibc, 8MB default stack):
$ ./thread-test-256k
Thread created successfully
Thread function executed
Stack used: ~260 KB
✓ SUCCESS: Plenty of room in 8 MB stack
On Alpine (musl, 128 KB default stack):
$ docker exec alpine /tmp/thread-test-256k
Segmentation fault (core dumped)
✗ EXPECTED CRASH: 256 KB > 128 KB default stack
Fix: Using pthread_attr_setstacksize()
$ docker exec alpine /tmp/thread-test-256k-fixed
Thread created with 2 MB stack
Thread function executed
✓ SUCCESS: Explicit stack size works
Stack Size Summary:
┌─────────────┬──────────────┬──────────────┐
│ Size │ glibc Result │ musl Result │
├─────────────┼──────────────┼──────────────┤
│ 64 KB │ ✓ Pass │ ✓ Pass │
│ 128 KB │ ✓ Pass │ ✗ Crash* │
│ 256 KB │ ✓ Pass │ ✗ Crash │
│ 512 KB │ ✓ Pass │ ✗ Crash │
│ 1 MB │ ✓ Pass │ ✗ Crash │
└─────────────┴──────────────┴──────────────┘
* 128 KB is at the limit and may crash depending on overhead
└──────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────┐
│ Test 3: DNS Resolution │
├──────────────────────────────────────────────────────────────┤
Sequential DNS lookups (10 iterations, google.com):
glibc Results:
Lookup 1: 12.34 ms
Lookup 2: 0.45 ms (cached!)
Lookup 3: 0.38 ms (cached!)
...
Average: 1.82 ms
musl Results:
Lookup 1: 15.67 ms
Lookup 2: 14.23 ms (no cache)
Lookup 3: 13.89 ms (no cache)
...
Average: 14.12 ms
Analysis:
glibc caches DNS responses internally (unless using nscd)
musl relies on system-level caching
For high-traffic applications, consider adding dnsmasq
Concurrent DNS lookups (100 simultaneous):
glibc: 234 ms total (parallel queries)
musl: 1,847 ms total (sequential processing)
Note: musl's resolver is single-threaded by design.
For high-concurrency DNS, use a caching resolver like dnsmasq.
└──────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────┐
│ Test 4: Locale Handling │
├──────────────────────────────────────────────────────────────┤
Available locales:
glibc (Ubuntu):
C, C.UTF-8, en_US.UTF-8, fr_FR.UTF-8, de_DE.UTF-8,
ja_JP.UTF-8, zh_CN.UTF-8, ... (100+ locales)
musl (Alpine):
C, C.UTF-8, POSIX
String sorting with LC_COLLATE=en_US.UTF-8:
Input: ["apple", "Banana", "cherry"]
glibc: ["apple", "Banana", "cherry"] (case-insensitive sort)
musl: ["Banana", "apple", "cherry"] (ASCII order, B < a)
Character classification (isupper('É')):
glibc: true (locale-aware)
musl: true (UTF-8 aware, same result in this case)
Recommendation:
Don't rely on locale-specific sorting in portable code.
Use explicit collation libraries if needed (ICU).
└──────────────────────────────────────────────────────────────┘
╔══════════════════════════════════════════════════════════════╗
║ Summary ║
╠══════════════════════════════════════════════════════════════╣
║ Tests Run: 4 ║
║ Expected Failures Demonstrated: 5 ║
║ Fixes Demonstrated: 4 ║
║ ║
║ Key Takeaways: ║
║ 1. Compile for musl when targeting Alpine ║
║ 2. Set explicit thread stack sizes ║
║ 3. Consider DNS caching for high-traffic applications ║
║ 4. Don't rely on locale-specific behavior ║
╚══════════════════════════════════════════════════════════════╝
Real World Outcome
After completing this project, you will be able to:
- Diagnose glibc/musl issues instantly when you see “not found” errors for existing binaries
- Choose the right deployment strategy (static linking vs recompilation vs compatibility layer)
- Prevent thread crashes by setting appropriate stack sizes
- Optimize DNS performance in Alpine-based applications
- Write portable C code that works on both glibc and musl systems
Solution Architecture
High-Level Design
┌─────────────────────────────────────────────────────────────────────┐
│ Test Suite Architecture │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ run-tests.sh │ │
│ │ (Main Test Runner) │ │
│ └───────────────────────────┬─────────────────────────────────┘ │
│ │ │
│ ┌───────────────────┼───────────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │
│ │ test-binary │ │ test-thread │ │ test-dns │ │
│ │ compatibility │ │ stack │ │ resolution │ │
│ └───────┬───────┘ └───────┬───────┘ └───────┬───────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │
│ │ test-locale │ │ test-fixes │ │ Dockerfile │ │
│ │ handling │ │ demonstrator │ │ (environments)│ │
│ └───────────────┘ └───────────────┘ └───────────────┘ │
│ │
│ Docker Environments: │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Ubuntu:latest │ │ Alpine:latest │ │
│ │ (glibc) │◄────────►│ (musl) │ │
│ └─────────────────┘ └─────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
Key Components
| Component | Purpose | Key Files |
|---|---|---|
| Test Runner | Orchestrates all tests | run-tests.sh |
| Binary Compat Test | Demonstrates linking issues | hello.c, test-binary.sh |
| Thread Stack Test | Shows stack overflow | thread-test.c |
| DNS Test | Benchmarks DNS resolution | dns-test.c |
| Locale Test | Shows locale differences | locale-test.c |
| Docker Setup | Provides both environments | Dockerfile.ubuntu, Dockerfile.alpine |
Data Structures
Thread Test Configuration:
struct thread_test_config {
size_t stack_size; // Bytes to allocate on stack
size_t explicit_stack; // Explicitly set stack size (0 = default)
int expected_result; // 0 = expect crash, 1 = expect success
};
DNS Test Results:
struct dns_result {
char hostname[256];
double latency_ms;
int success;
char error[256];
};
Algorithm Overview
Binary Compatibility Test Flow:
- Compile simple C program on Ubuntu (glibc)
- Copy binary to Alpine container
- Attempt execution (expect failure)
- Analyze with
lddto show dependencies - Demonstrate fix methods (static link, gcompat, recompile)
Thread Stack Test Flow:
- Create thread with varying stack usage
- Use
pthread_attr_setstacksize()for explicit sizing - Compare results between musl and glibc
- Report which sizes pass/fail
Implementation Guide
Development Environment Setup
# Create project directory
mkdir -p musl-compatibility-lab
cd musl-compatibility-lab
# Create directory structure
mkdir -p src tests docker results
# Required tools (on Ubuntu host)
sudo apt update
sudo apt install -y build-essential docker.io
# Start Docker
sudo systemctl start docker
sudo usermod -aG docker $USER
# Log out and back in for group membership
# Pull required images
docker pull ubuntu:latest
docker pull alpine:latest
# Verify
docker run --rm ubuntu:latest cat /etc/os-release | head -2
docker run --rm alpine:latest cat /etc/os-release | head -2
Project Structure
musl-compatibility-lab/
├── run-tests.sh # Main test runner
├── docker/
│ ├── Dockerfile.ubuntu # Ubuntu build environment
│ └── Dockerfile.alpine # Alpine test environment
├── src/
│ ├── hello.c # Simple binary for compatibility test
│ ├── thread-test.c # Thread stack test
│ ├── thread-test-fixed.c # Thread test with explicit stack
│ ├── dns-test.c # DNS resolution benchmark
│ └── locale-test.c # Locale handling test
├── tests/
│ ├── test-binary-compat.sh # Binary compatibility tests
│ ├── test-thread-stack.sh # Thread stack tests
│ ├── test-dns.sh # DNS tests
│ └── test-locale.sh # Locale tests
└── results/
└── (test output files)
The Core Question You’re Answering
“Why do binaries that work perfectly on Ubuntu fail mysteriously on Alpine, and how do I fix it?”
This project answers this question by demonstrating:
- The fundamental architectural difference (musl vs glibc)
- How this difference manifests (linking errors, crashes, behavior differences)
- Multiple strategies for resolution
Concepts You Must Understand First
Before starting, verify you understand:
- Dynamic Linking: What is a shared library? What does
lddshow?- Book: TLPI Ch. 41 “Fundamentals of Shared Libraries”
- The C Compilation Process: Preprocessing → Compilation → Assembly → Linking
- Book: TLPI Ch. 41, sections 41.1-41.4
- POSIX Threads Basics:
pthread_create(),pthread_join(), thread attributes- Book: TLPI Ch. 29 “Threads: Introduction”
- DNS Resolution: What is
getaddrinfo()? How does/etc/resolv.confwork?- Book: TLPI Ch. 59 “Sockets: Internet Domains”
Questions to Guide Your Design
Binary Compatibility:
- What does the
lddoutput tell you about a binary’s dependencies? - What is the ELF interpreter and why does it matter?
- When is static linking appropriate? When is it not?
Thread Stack:
- How does the OS allocate stack space for threads?
- What happens when a thread exceeds its stack?
- How do you determine the actual stack usage of a thread?
DNS:
- What’s the difference between
gethostbyname()andgetaddrinfo()? - How does DNS caching work at the application level vs system level?
- What happens when a DNS query times out?
Locale:
- What does
setlocale()actually do? - How does locale affect string comparison (
strcmpvsstrcoll)? - What is the difference between byte strings and wide strings?
Thinking Exercise
Before writing any code, trace through this scenario:
Scenario: You have a Go program that uses CGO to call a C library. The program works on your Ubuntu laptop but crashes with SIGSEGV when running in an Alpine-based Docker container.
Questions to answer:
- What is CGO and why does it matter for this problem?
- If you set
CGO_ENABLED=0, what changes about the compiled binary? - If the crash only happens under high load with many goroutines, what specific musl behavior might be the cause?
- How would you diagnose whether the crash is a stack overflow vs a linking issue?
- What would you look for in the core dump (if you could get one)?
Hints in Layers
Hint 1: Starting Point
Binary Compatibility:
Start with the simplest possible C program (hello.c). Compile it dynamically on Ubuntu, then try to run it on Alpine. The error message tells you exactly what’s missing.
Thread Stack: Create a thread that allocates a local array larger than 128 KB. Compare behavior between Ubuntu and Alpine.
DNS:
Use gethostbyname() in a loop with timing. Run the same code on both systems and compare results.
Hint 2: Next Level
Binary Compatibility:
Use ldd to inspect the binary. Notice it wants /lib64/ld-linux-x86-64.so.2 (glibc’s dynamic linker). Alpine has /lib/ld-musl-x86_64.so.1. They’re incompatible.
Thread Stack:
After demonstrating the crash, use pthread_attr_t to explicitly set the stack size:
pthread_attr_t attr;
pthread_attr_init(&attr);
pthread_attr_setstacksize(&attr, 2 * 1024 * 1024); // 2 MB
pthread_create(&thread, &attr, func, NULL);
DNS: After measuring sequential performance, try concurrent lookups. glibc will parallelize; musl will serialize. This is by design.
Hint 3: Technical Details
Binary Compatibility - Three Solutions:
# Solution 1: Static linking (best for Go, Rust)
CGO_ENABLED=0 go build -o myapp
RUSTFLAGS='-C target-feature=+crt-static' cargo build --release --target x86_64-unknown-linux-musl
# Solution 2: Recompile on Alpine
docker run --rm -v $(pwd):/src alpine:latest sh -c "
apk add --no-cache build-base
gcc -o /src/hello-musl /src/hello.c
"
# Solution 3: Compatibility layer (last resort)
docker run --rm alpine:latest sh -c "
apk add --no-cache gcompat
./glibc-binary
"
Thread Stack - Measuring Actual Usage:
#include <pthread.h>
void* thread_func(void* arg) {
pthread_attr_t attr;
size_t stack_size;
pthread_getattr_np(pthread_self(), &attr);
pthread_attr_getstacksize(&attr, &stack_size);
printf("Actual stack size: %zu bytes\n", stack_size);
return NULL;
}
Hint 4: Tools and Debugging
Debugging Binary Compatibility:
# Check what libc a binary was linked against
file ./mybinary
# ELF 64-bit LSB executable, x86-64, dynamically linked,
# interpreter /lib64/ld-linux-x86-64.so.2
# On Alpine, check what musl provides
ls -la /lib/ld-musl-*
# /lib/ld-musl-x86_64.so.1
Debugging Stack Issues:
# Run with stack trace on crash
docker run --rm alpine:latest sh -c "
apk add --no-cache gdb
gdb -ex run -ex bt ./thread-test
"
# Check default stack size
docker run --rm alpine:latest sh -c "
ulimit -s # Typically 8192 (8 MB) but that's the main thread!
"
Debugging DNS:
# Check resolver configuration
docker run --rm alpine:latest cat /etc/resolv.conf
# Test DNS with timing
docker run --rm alpine:latest sh -c "
time nslookup google.com
"
The Interview Questions They’ll Ask
- “Why don’t glibc binaries run on Alpine?”
- Expected answer: Different C library (musl vs glibc), different dynamic linker, incompatible ABI
- Deep answer: Discuss ELF interpreter, symbol versioning, ABI differences
- “How do you handle the smaller default thread stack in musl?”
- Expected answer: Use
pthread_attr_setstacksize() - Deep answer: Discuss why musl chose 128 KB, when smaller stacks are appropriate, how to measure actual usage
- Expected answer: Use
- “What are the DNS differences between musl and glibc?”
- Expected answer: musl is simpler, single-threaded resolver, no internal caching
- Deep answer: Discuss getaddrinfo implementation, search domains, how to add caching
- “When would you choose static linking vs recompilation vs gcompat?”
- Static linking: Simple binaries, Go/Rust, no NSS needed
- Recompilation: Best compatibility, have source code, clean solution
- gcompat: Third-party binaries, temporary fix, testing
- “How would you debug a crash that only happens on Alpine?”
- Check if it’s a linking issue (ldd, file)
- Check if it’s stack size (thread-related crash)
- Check if it’s locale/character handling
- Use gdb in container, check dmesg for segfaults
Books That Will Help
| Topic | Book | Chapter |
|---|---|---|
| Shared Libraries | “The Linux Programming Interface” by Kerrisk | Ch. 41: Fundamentals of Shared Libraries |
| Dynamic Linking | “The Linux Programming Interface” by Kerrisk | Ch. 42: Advanced Features of Shared Libraries |
| POSIX Threads | “The Linux Programming Interface” by Kerrisk | Ch. 29-33: Threads |
| DNS/Networking | “The Linux Programming Interface” by Kerrisk | Ch. 59: Sockets: Internet Domains |
| ELF Format | “Linkers and Loaders” by John Levine | Ch. 3: Object Files |
| C Runtime | “Expert C Programming” by Peter van der Linden | Ch. 6: Runtime Data Structures |
Testing Strategy
Unit Tests
# Test 1: Binary compilation works
gcc -o hello hello.c
[ -f hello ] && echo "PASS" || echo "FAIL"
# Test 2: Binary runs on source platform
./hello | grep -q "Hello" && echo "PASS" || echo "FAIL"
# Test 3: Thread test compiles
gcc -pthread -o thread-test thread-test.c
[ -f thread-test ] && echo "PASS" || echo "FAIL"
Integration Tests
# Test: glibc binary fails on Alpine (expected)
docker run --rm -v $(pwd):/app alpine:latest /app/hello 2>&1 | \
grep -q "not found" && echo "EXPECTED FAILURE" || echo "UNEXPECTED SUCCESS"
# Test: musl-compiled binary works on Alpine
docker run --rm -v $(pwd):/app alpine:latest /app/hello-musl 2>&1 | \
grep -q "Hello" && echo "PASS" || echo "FAIL"
Common Pitfalls & Debugging
Problem 1: “not found” for existing binary
Symptom:
$ ./mybinary
/bin/sh: ./mybinary: not found
Root Cause: The binary was linked against glibc’s dynamic linker, which doesn’t exist on Alpine.
Fix:
# Verify with file command
file ./mybinary
# If it shows "interpreter /lib64/ld-linux-x86-64.so.2" -> glibc binary
# Solutions:
1. Static link: CGO_ENABLED=0 go build
2. Recompile on Alpine: docker run -v... alpine gcc -o mybinary mybinary.c
3. Add gcompat: apk add gcompat
Verification:
# Check that binary now works
docker run --rm -v $(pwd):/app alpine:latest /app/mybinary
Problem 2: Random crashes in threaded code
Symptom: Program crashes with SIGSEGV but only when running multiple threads, and only on Alpine.
Root Cause: Thread stack overflow due to musl’s 128 KB default.
Fix:
pthread_attr_t attr;
pthread_attr_init(&attr);
pthread_attr_setstacksize(&attr, 2 * 1024 * 1024); // 2 MB
pthread_create(&thread, &attr, thread_func, NULL);
pthread_attr_destroy(&attr);
Verification:
# Run thread test on Alpine
docker run --rm -v $(pwd):/app alpine:latest /app/thread-test-fixed
# Should complete without crash
Problem 3: Slow DNS under load
Symptom: Application becomes slow under high concurrency, DNS lookups take much longer than expected.
Root Cause: musl’s resolver is single-threaded and doesn’t cache.
Fix:
# Option 1: Add local DNS cache
apk add dnsmasq
echo "server=8.8.8.8" >> /etc/dnsmasq.conf
echo "nameserver 127.0.0.1" > /etc/resolv.conf
# Option 2: Use application-level caching
# Implement DNS cache in your application
Verification:
# Benchmark before and after
./dns-benchmark
Problem 4: String sorting differs
Symptom: Sorted results are different between Ubuntu and Alpine.
Root Cause: musl doesn’t implement locale-specific collation.
Fix:
// Don't rely on strcoll() for portable code
// Either use strcmp() (byte comparison)
// Or use a library like ICU for locale-aware sorting
Verification:
# Test on both systems
./locale-test # Compare output
Extensions & Challenges
Challenge 1: Cross-Compilation
Set up a cross-compilation environment to build musl binaries on Ubuntu without using Docker.
# Install musl cross-compiler
sudo apt install musl-tools
# Compile for musl
musl-gcc -o hello-musl hello.c
Challenge 2: Performance Benchmark
Create a comprehensive benchmark comparing:
- Memory allocator performance (malloc/free)
- Thread creation overhead
- DNS resolution latency
- Math library performance
Challenge 3: Automatic Compatibility Checker
Build a tool that analyzes an ELF binary and reports potential musl compatibility issues:
- Checks for glibc-specific symbols
- Warns about thread stack usage patterns
- Detects locale-dependent code
Challenge 4: Go/Rust/Python Integration
Test the same scenarios with:
- Go programs with and without CGO
- Rust programs with different target triples
- Python programs with native extensions
Real-World Connections
Case Study 1: Node.js on Alpine
The Node.js Alpine images have specific considerations:
- Native addons must be compiled for musl
- DNS resolution behavior affects http requests
- Thread pool size affects performance
Case Study 2: Python with NumPy/SciPy
NumPy and SciPy use optimized C code:
- BLAS/LAPACK must be musl-compatible
- Wheel packages are glibc-specific
- Alpine needs to compile from source
Case Study 3: Kubernetes CrashLoopBackOff
Many “CrashLoopBackOff” issues in Kubernetes are musl-related:
- glibc binary in Alpine container
- Thread stack overflow under load
- DNS timeout issues
Resources
Official Documentation
Technical Articles
Tools
ldd- Print shared library dependenciesfile- Determine file type (shows ELF interpreter)readelf- Display ELF file informationstrace- Trace system calls
Self-Assessment Checklist
After completing this project, you should be able to:
- Explain why glibc binaries don’t run on Alpine
- Use
lddandfileto diagnose binary compatibility - Choose between static linking, recompilation, and gcompat
- Set explicit thread stack sizes with pthread_attr_setstacksize
- Explain musl’s DNS resolver design and its implications
- Write C code that works on both musl and glibc
- Debug container crashes related to libc differences
- Configure DNS caching for musl-based systems
Submission / Completion Criteria
Your project is complete when:
- All test programs compile and run on both Ubuntu and Alpine
- Binary compatibility test correctly demonstrates failure and all three fixes
- Thread stack test shows crash on musl and success with explicit sizing
- DNS test shows measurable difference in caching behavior
- Locale test demonstrates sorting differences
- Documentation explains each test’s purpose and results
- You can explain each difference to another developer
Understanding the musl vs glibc difference is foundational to working with Alpine Linux. Once you internalize these concepts, you’ll instantly recognize and fix issues that stump other developers.