Project 2: Configuration Management System

Project 2: Configuration Management System

Build a type-safe configuration system that loads settings from environment variables, .env files, config files, and CLI arguments with full validation and documentation.


Learning Objectives

By completing this project, you will:

  1. Master pydantic-settings - Understand how it extends Pydantic for configuration management
  2. Handle multiple configuration sources - Learn source precedence and how settings cascade
  3. Secure sensitive data - Use SecretStr to prevent accidental secret exposure
  4. Build nested configuration - Create hierarchical settings with sub-configurations
  5. Generate configuration documentation - Auto-document all available settings
  6. Implement environment-specific configs - Handle dev/staging/production configurations

Theoretical Foundation

The 12-Factor App Configuration Principle

The 12-Factor App methodology defines a best practice for handling configuration:

โ€œStore config in the environmentโ€

This means:

  • No hardcoded values - Configuration should never be committed to code
  • Environment variables - The primary mechanism for passing config
  • Environment parity - Same code runs in dev, staging, production with different configs
  • No code changes for config changes - Reconfigure by changing environment, not code
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚                    Configuration Flow                            โ”‚
โ”‚                                                                  โ”‚
โ”‚   โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”   โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”   โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”   โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”    โ”‚
โ”‚   โ”‚ Default  โ”‚   โ”‚  .env    โ”‚   โ”‚ Environ  โ”‚   โ”‚   CLI    โ”‚    โ”‚
โ”‚   โ”‚ Values   โ”‚   โ”‚  File    โ”‚   โ”‚  Vars    โ”‚   โ”‚  Args    โ”‚    โ”‚
โ”‚   โ””โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”˜   โ””โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”˜   โ””โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”˜   โ””โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”˜    โ”‚
โ”‚        โ”‚              โ”‚              โ”‚              โ”‚           โ”‚
โ”‚        โ”‚   Lowest     โ”‚              โ”‚              โ”‚  Highest  โ”‚
โ”‚        โ”‚  Priority    โ”‚              โ”‚              โ”‚  Priority โ”‚
โ”‚        โ”‚              โ”‚              โ”‚              โ”‚           โ”‚
โ”‚        โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜           โ”‚
โ”‚                              โ”‚                                   โ”‚
โ”‚                              โ–ผ                                   โ”‚
โ”‚                    โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”                         โ”‚
โ”‚                    โ”‚ Pydantic Settingsโ”‚                         โ”‚
โ”‚                    โ”‚   (Validated)    โ”‚                         โ”‚
โ”‚                    โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜                         โ”‚
โ”‚                              โ”‚                                   โ”‚
โ”‚                              โ–ผ                                   โ”‚
โ”‚                    โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”                         โ”‚
โ”‚                    โ”‚  Your Applicationโ”‚                         โ”‚
โ”‚                    โ”‚  (Type-Safe!)    โ”‚                         โ”‚
โ”‚                    โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜                         โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

Configuration Flow

Why Type-Safe Configuration?

Traditional configuration approaches are error-prone:

# Traditional approach - many problems!
import os

DATABASE_URL = os.getenv("DATABASE_URL")  # Could be None!
DEBUG = os.getenv("DEBUG")  # "true" or "True" or "1"? It's a string!
PORT = int(os.getenv("PORT", 8000))  # Crashes if PORT="abc"
MAX_CONNECTIONS = os.getenv("MAX_CONN")  # Typo goes unnoticed!

Problems with this approach:

  1. No validation - Invalid values arenโ€™t caught until runtime
  2. Type confusion - Everything is a string
  3. Silent failures - Missing required values return None
  4. No documentation - What settings exist? What are valid values?
  5. Typos - Misspelled variable names silently fail

Pydantic Settings solves all of these:

from pydantic_settings import BaseSettings
from pydantic import Field, PostgresDsn

class Settings(BaseSettings):
    database_url: PostgresDsn  # Validated URL!
    debug: bool = False  # Automatically parsed from "true", "1", etc.
    port: int = Field(default=8000, ge=1, le=65535)  # Range validation!
    max_connections: int = Field(default=10, ge=1)  # Required name match!

settings = Settings()  # Validates on instantiation!

How pydantic-settings Works

pydantic-settings extends Pydanticโ€™s BaseModel with environment variable loading:

from pydantic_settings import BaseSettings, SettingsConfigDict

class AppSettings(BaseSettings):
    app_name: str
    debug: bool = False

    model_config = SettingsConfigDict(
        env_file='.env',
        env_file_encoding='utf-8',
        case_sensitive=False,  # APP_NAME or app_name both work
    )

Loading Priority (lowest to highest):

  1. Default values in the class definition
  2. Environment file (.env)
  3. Environment variables (actual OS environment)
  4. Init arguments (when instantiating the class)

This means environment variables override .env file values, which override defaults.

Understanding Settings Sources

pydantic-settings uses โ€œsourcesโ€ to load values:

from pydantic_settings import (
    BaseSettings,
    SettingsConfigDict,
    PydanticBaseSettingsSource,
)

class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file='.env',
        env_nested_delimiter='__',  # For nested settings
    )

    @classmethod
    def settings_customise_sources(
        cls,
        settings_cls: type[BaseSettings],
        init_settings: PydanticBaseSettingsSource,
        env_settings: PydanticBaseSettingsSource,
        dotenv_settings: PydanticBaseSettingsSource,
        file_secret_settings: PydanticBaseSettingsSource,
    ) -> tuple[PydanticBaseSettingsSource, ...]:
        # Customize source order (priority)
        return (
            init_settings,      # Highest priority
            env_settings,       # Environment variables
            dotenv_settings,    # .env file
            file_secret_settings,  # Secret files (Docker secrets)
        )

SecretStr: Protecting Sensitive Data

One of the biggest configuration mistakes is accidentally logging secrets:

# DANGER: This logs your password!
print(f"Connecting to {settings.database_password}")
# Or in error messages, logs, etc.

SecretStr prevents this:

from pydantic import SecretStr

class Settings(BaseSettings):
    database_password: SecretStr

settings = Settings()

# Safe operations
print(settings.database_password)  # Output: **********
str(settings.database_password)    # Output: **********
repr(settings.database_password)   # Output: SecretStr('**********')

# Explicit reveal (when you actually need the value)
password = settings.database_password.get_secret_value()

SecretStr also works in:

  • JSON serialization (excluded by default)
  • Logging
  • Error messages
  • Debugger views

Nested Configuration

Real applications have hierarchical configuration:

from pydantic_settings import BaseSettings, SettingsConfigDict
from pydantic import Field

class DatabaseSettings(BaseSettings):
    host: str = "localhost"
    port: int = 5432
    name: str
    user: str
    password: SecretStr

    model_config = SettingsConfigDict(env_prefix='DB_')

class RedisSettings(BaseSettings):
    url: str = "redis://localhost:6379"
    password: SecretStr | None = None

    model_config = SettingsConfigDict(env_prefix='REDIS_')

class AppSettings(BaseSettings):
    debug: bool = False
    secret_key: SecretStr

    # Nested settings
    database: DatabaseSettings = Field(default_factory=DatabaseSettings)
    redis: RedisSettings = Field(default_factory=RedisSettings)

    model_config = SettingsConfigDict(
        env_file='.env',
        env_nested_delimiter='__',  # Enables DATABASE__HOST syntax
    )

Environment variables for nested settings:

# Using prefix (DatabaseSettings has env_prefix='DB_')
DB_HOST=postgres.example.com
DB_PORT=5432
DB_NAME=myapp
DB_USER=admin
DB_PASSWORD=secret

# OR using nested delimiter
DATABASE__HOST=postgres.example.com
DATABASE__PORT=5432

Complex Types in Settings

Pydantic Settings handles various types:

from pydantic_settings import BaseSettings
from pydantic import Field, HttpUrl, EmailStr
from typing import List, Dict
from datetime import timedelta

class Settings(BaseSettings):
    # URLs are validated
    api_url: HttpUrl

    # Email validation
    admin_email: EmailStr

    # Lists from JSON or comma-separated
    allowed_hosts: List[str] = Field(default=["localhost"])
    # As env: ALLOWED_HOSTS='["host1.com", "host2.com"]'
    # Or: ALLOWED_HOSTS=host1.com,host2.com (with custom validator)

    # Dicts from JSON
    feature_flags: Dict[str, bool] = Field(default_factory=dict)
    # As env: FEATURE_FLAGS='{"new_ui": true, "beta": false}'

    # Timedeltas
    cache_ttl: timedelta = timedelta(hours=1)
    # As env: CACHE_TTL=3600 (seconds)

    # Enums
    log_level: LogLevel = LogLevel.INFO

Project Specification

Functional Requirements

Build a configuration management system and CLI tool that:

  1. Manages application settings
    • Define settings using Pydantic models
    • Load from multiple sources with proper precedence
    • Validate all settings on startup
  2. Provides a CLI interface
    • Show current configuration (with masked secrets)
    • Validate configuration without starting the app
    • Export configuration to various formats
    • Generate documentation
  3. Supports multiple environments
    • Development, staging, production profiles
    • Environment-specific .env files
    • Override mechanism
  4. Handles secrets safely
    • Never log or display secret values
    • Support Docker secrets
    • Warn about insecure configurations

CLI Interface

# Show current configuration
$ config-manager show
โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ Application Configuration                                   โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ Environment: production                                     โ”‚
โ”‚                                                             โ”‚
โ”‚ app:                                                        โ”‚
โ”‚   name: MyApp                                              โ”‚
โ”‚   debug: false                                             โ”‚
โ”‚   secret_key: ******** (SecretStr)                         โ”‚
โ”‚                                                             โ”‚
โ”‚ database:                                                   โ”‚
โ”‚   host: postgres.example.com                               โ”‚
โ”‚   port: 5432                                               โ”‚
โ”‚   name: myapp_prod                                         โ”‚
โ”‚   user: app_user                                           โ”‚
โ”‚   password: ******** (SecretStr)                           โ”‚
โ”‚   pool_size: 20                                            โ”‚
โ”‚                                                             โ”‚
โ”‚ redis:                                                      โ”‚
โ”‚   url: redis://redis.example.com:6379                      โ”‚
โ”‚   password: ******** (SecretStr)                           โ”‚
โ”‚                                                             โ”‚
โ”‚ logging:                                                    โ”‚
โ”‚   level: INFO                                              โ”‚
โ”‚   format: json                                             โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ

# Validate configuration
$ config-manager validate
โœ“ All required settings present
โœ“ Database URL is valid
โœ“ Redis connection successful
โœ“ Secret key has sufficient entropy
โœ“ Configuration is valid for production

# With missing required settings
$ config-manager validate
โœ— Validation failed:
  โ”œโ”€โ”€ SECRET_KEY: Field required (no default)
  โ”œโ”€โ”€ DB_PASSWORD: Field required (no default)
  โ””โ”€โ”€ Suggestion: Create a .env file or set environment variables

# Export configuration
$ config-manager export --format=json --include-secrets=false > config.json
$ config-manager export --format=yaml
$ config-manager export --format=env > .env.example

# Generate documentation
$ config-manager docs
# Application Configuration

## Environment Variables

| Variable | Type | Required | Default | Description |
|----------|------|----------|---------|-------------|
| APP_NAME | str | No | "MyApp" | Application name |
| DEBUG | bool | No | false | Enable debug mode |
| SECRET_KEY | SecretStr | Yes | - | Application secret key |
| DB_HOST | str | No | "localhost" | Database host |
...

# Show specific section
$ config-manager show database
$ config-manager show --env=staging

# Diff between environments
$ config-manager diff development production

Settings Structure

# config/settings.py
from pydantic_settings import BaseSettings, SettingsConfigDict
from pydantic import Field, SecretStr, PostgresDsn, RedisDsn
from typing import Optional, List, Literal
from enum import Enum

class Environment(str, Enum):
    development = "development"
    staging = "staging"
    production = "production"

class LogLevel(str, Enum):
    DEBUG = "DEBUG"
    INFO = "INFO"
    WARNING = "WARNING"
    ERROR = "ERROR"

class DatabaseSettings(BaseSettings):
    """Database connection settings."""
    host: str = Field(
        default="localhost",
        description="Database server hostname"
    )
    port: int = Field(
        default=5432,
        ge=1,
        le=65535,
        description="Database server port"
    )
    name: str = Field(
        description="Database name"
    )
    user: str = Field(
        description="Database username"
    )
    password: SecretStr = Field(
        description="Database password"
    )
    pool_size: int = Field(
        default=5,
        ge=1,
        le=100,
        description="Connection pool size"
    )

    model_config = SettingsConfigDict(env_prefix='DB_')

    @property
    def url(self) -> str:
        """Construct database URL."""
        password = self.password.get_secret_value()
        return f"postgresql://{self.user}:{password}@{self.host}:{self.port}/{self.name}"

class RedisSettings(BaseSettings):
    """Redis connection settings."""
    url: RedisDsn = Field(
        default="redis://localhost:6379",
        description="Redis connection URL"
    )
    password: Optional[SecretStr] = Field(
        default=None,
        description="Redis password (if required)"
    )
    db: int = Field(
        default=0,
        ge=0,
        le=15,
        description="Redis database number"
    )

    model_config = SettingsConfigDict(env_prefix='REDIS_')

class LoggingSettings(BaseSettings):
    """Logging configuration."""
    level: LogLevel = Field(
        default=LogLevel.INFO,
        description="Log level"
    )
    format: Literal["json", "text"] = Field(
        default="text",
        description="Log output format"
    )
    file: Optional[str] = Field(
        default=None,
        description="Log file path (optional)"
    )

    model_config = SettingsConfigDict(env_prefix='LOG_')

class AppSettings(BaseSettings):
    """Main application settings."""
    name: str = Field(
        default="MyApp",
        description="Application name"
    )
    environment: Environment = Field(
        default=Environment.development,
        description="Deployment environment"
    )
    debug: bool = Field(
        default=False,
        description="Enable debug mode (never in production!)"
    )
    secret_key: SecretStr = Field(
        description="Secret key for cryptographic operations",
        min_length=32
    )
    allowed_hosts: List[str] = Field(
        default=["localhost", "127.0.0.1"],
        description="Allowed host headers"
    )

    # Nested settings
    database: DatabaseSettings = Field(default_factory=DatabaseSettings)
    redis: RedisSettings = Field(default_factory=RedisSettings)
    logging: LoggingSettings = Field(default_factory=LoggingSettings)

    model_config = SettingsConfigDict(
        env_file='.env',
        env_file_encoding='utf-8',
        env_nested_delimiter='__',
        extra='ignore',  # Ignore unknown env vars
    )

    def validate_for_production(self) -> List[str]:
        """Additional production-specific validation."""
        warnings = []

        if self.environment == Environment.production:
            if self.debug:
                warnings.append("DEBUG is enabled in production!")
            if "localhost" in self.allowed_hosts:
                warnings.append("localhost in ALLOWED_HOSTS for production")
            if len(self.secret_key.get_secret_value()) < 50:
                warnings.append("SECRET_KEY should be at least 50 characters")

        return warnings

Solution Architecture

Component Design

โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚                         CLI Layer                                โ”‚
โ”‚                     (Typer/Click App)                           โ”‚
โ”‚  Commands: show, validate, export, docs, diff                   โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                              โ”‚
                              โ–ผ
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚                    Settings Manager                              โ”‚
โ”‚  - Load settings from various sources                           โ”‚
โ”‚  - Handle environment-specific overrides                         โ”‚
โ”‚  - Provide access to current configuration                       โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                              โ”‚
            โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
            โ–ผ                 โ–ผ                 โ–ผ
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚   Validators      โ”‚ โ”‚   Formatters      โ”‚ โ”‚   Exporters       โ”‚
โ”‚  - Schema check   โ”‚ โ”‚  - Console table  โ”‚ โ”‚  - JSON           โ”‚
โ”‚  - Connection testโ”‚ โ”‚  - Masked secrets โ”‚ โ”‚  - YAML           โ”‚
โ”‚  - Security audit โ”‚ โ”‚  - Tree view      โ”‚ โ”‚  - .env format    โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                              โ”‚
                              โ–ผ
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚                    Documentation Generator                       โ”‚
โ”‚  - Extract Field descriptions                                    โ”‚
โ”‚  - Generate markdown tables                                      โ”‚
โ”‚  - Include examples and defaults                                 โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

Component Design

Key Design Patterns

  1. Singleton Settings: Load configuration once, access globally
  2. Environment Overlay: Base config + environment-specific overrides
  3. Fail-Fast Validation: Catch config errors at startup
  4. Secret Masking: Never expose secrets in any output

Settings Loading Strategy

# config/loader.py
from functools import lru_cache
from pathlib import Path
from typing import Optional

@lru_cache
def get_settings(environment: Optional[str] = None) -> AppSettings:
    """
    Load and cache settings.

    Priority:
    1. .env.{environment}.local (ignored by git)
    2. .env.{environment}
    3. .env.local (ignored by git)
    4. .env
    5. Environment variables
    """
    env_files = []

    if environment:
        local_env = Path(f".env.{environment}.local")
        if local_env.exists():
            env_files.append(local_env)

        env_file = Path(f".env.{environment}")
        if env_file.exists():
            env_files.append(env_file)

    local_env = Path(".env.local")
    if local_env.exists():
        env_files.append(local_env)

    base_env = Path(".env")
    if base_env.exists():
        env_files.append(base_env)

    # Load with file priority
    return AppSettings(_env_file=env_files)

Phased Implementation Guide

Phase 1: Basic Settings Class (1-2 hours)

Goal: Create a working settings class with pydantic-settings.

  1. Install dependencies:
    pip install pydantic pydantic-settings python-dotenv
    
  2. Create basic settings:
    from pydantic_settings import BaseSettings
    
    class Settings(BaseSettings):
        app_name: str = "MyApp"
        debug: bool = False
    
  3. Test loading from environment:
    DEBUG=true python -c "from settings import Settings; print(Settings().debug)"
    

Checkpoint: Settings load from environment variables.

Phase 2: Nested Settings (1-2 hours)

Goal: Implement hierarchical configuration.

  1. Create database and redis settings classes
  2. Add env_prefix to each
  3. Compose into main AppSettings
  4. Test with environment variables

Checkpoint: DB_HOST=x python -c "print(Settings().database.host)" works.

Phase 3: Secret Handling (1 hour)

Goal: Implement secure secret handling.

  1. Add SecretStr fields for passwords, keys
  2. Test that printing doesnโ€™t reveal secrets
  3. Implement get_secret_value() for actual usage

Checkpoint: print(settings) shows ******** for secrets.

Phase 4: CLI Tool (2-3 hours)

Goal: Build the config-manager CLI.

  1. Set up Typer/Click application
  2. Implement show command with Rich formatting
  3. Implement validate command
  4. Add --env option for environment selection

Checkpoint: config-manager show displays formatted config.

Phase 5: Export and Documentation (2 hours)

Goal: Generate exports and docs.

  1. Implement JSON/YAML export (with secret exclusion)
  2. Implement .env.example generation
  3. Build documentation generator from Field metadata

Checkpoint: config-manager docs generates markdown.

Phase 6: Advanced Features (2-3 hours)

Goal: Add production-ready features.

  1. Implement environment-specific .env file loading
  2. Add production validation checks
  3. Implement config diff between environments
  4. Add connection testing for database/redis

Checkpoint: Full working configuration management system.


Testing Strategy

Unit Tests

# tests/test_settings.py
import os
import pytest
from config.settings import AppSettings, DatabaseSettings

def test_default_values():
    """Test that defaults are applied."""
    with pytest.MonkeyPatch.context() as mp:
        mp.setenv("SECRET_KEY", "a" * 32)
        mp.setenv("DB_NAME", "test")
        mp.setenv("DB_USER", "user")
        mp.setenv("DB_PASSWORD", "pass")

        settings = AppSettings()
        assert settings.debug is False
        assert settings.database.host == "localhost"

def test_env_override():
    """Test environment variable override."""
    with pytest.MonkeyPatch.context() as mp:
        mp.setenv("DEBUG", "true")
        mp.setenv("DB_HOST", "custom-host")
        # ... other required vars

        settings = AppSettings()
        assert settings.debug is True
        assert settings.database.host == "custom-host"

def test_secret_masking():
    """Test that secrets are masked in string output."""
    with pytest.MonkeyPatch.context() as mp:
        mp.setenv("SECRET_KEY", "super-secret-value")
        # ... other required vars

        settings = AppSettings()
        string_repr = str(settings)
        assert "super-secret-value" not in string_repr

def test_missing_required_field():
    """Test that missing required fields raise error."""
    with pytest.MonkeyPatch.context() as mp:
        mp.delenv("SECRET_KEY", raising=False)

        with pytest.raises(ValidationError) as exc:
            AppSettings()
        assert "SECRET_KEY" in str(exc.value)

def test_production_validation():
    """Test production-specific validation."""
    with pytest.MonkeyPatch.context() as mp:
        mp.setenv("ENVIRONMENT", "production")
        mp.setenv("DEBUG", "true")
        # ... other vars

        settings = AppSettings()
        warnings = settings.validate_for_production()
        assert "DEBUG is enabled in production" in warnings

Integration Tests

# tests/test_cli.py
from typer.testing import CliRunner
from config.cli import app

runner = CliRunner()

def test_show_command():
    result = runner.invoke(app, ["show"])
    assert result.exit_code == 0
    assert "Application Configuration" in result.output

def test_validate_success():
    result = runner.invoke(app, ["validate"])
    assert result.exit_code == 0
    assert "valid" in result.output.lower()

def test_export_json():
    result = runner.invoke(app, ["export", "--format", "json"])
    assert result.exit_code == 0
    import json
    data = json.loads(result.output)
    assert "database" in data

def test_export_excludes_secrets():
    result = runner.invoke(app, ["export", "--format", "json"])
    assert "password" not in result.output.lower()

Common Pitfalls and Debugging

Pitfall 1: Case Sensitivity

Problem: DB_HOST works but db_host doesnโ€™t.

Solution: Set case_sensitive=False in SettingsConfigDict:

model_config = SettingsConfigDict(
    case_sensitive=False  # DB_HOST and db_host both work
)

Pitfall 2: Nested Environment Variables

Problem: DATABASE__HOST not loading nested settings.

Solution: Enable nested delimiter:

model_config = SettingsConfigDict(
    env_nested_delimiter='__'
)

Also ensure the nested class doesnโ€™t have a conflicting env_prefix.

Pitfall 3: Type Coercion Confusion

Problem: DEBUG=0 sets debug to True (non-empty string is truthy).

Solution: Pydantic handles this correctly! DEBUG=0 โ†’ False, DEBUG=false โ†’ False.

But beware of:

  • DEBUG= (empty string) โ†’ May fail validation
  • DEBUG=no โ†’ False
  • DEBUG=yes โ†’ True

Pitfall 4: .env File Not Loading

Problem: Values in .env file are ignored.

Checklist:

  1. Is python-dotenv installed?
  2. Is env_file='.env' in SettingsConfigDict?
  3. Is the .env file in the correct directory (working directory)?
  4. Are environment variables set that override .env?

Pitfall 5: SecretStr in JSON Export

Problem: model_dump_json() includes secrets.

Solution: Use exclude or customize serialization:

settings.model_dump_json(exclude={'database': {'password'}})

# Or use a custom serializer
class Settings(BaseSettings):
    @model_serializer(mode='wrap')
    def serialize(self, handler):
        data = handler(self)
        # Recursively remove SecretStr values
        return remove_secrets(data)

Extensions and Challenges

Extension 1: Docker Secrets Support

Load secrets from Docker secret files:

from pydantic_settings import BaseSettings, SettingsConfigDict

class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        secrets_dir='/run/secrets'
    )

Docker secrets are files like /run/secrets/db_password containing the secret value.

Extension 2: Remote Configuration

Load configuration from a remote source:

import httpx
from pydantic_settings import BaseSettings, PydanticBaseSettingsSource

class RemoteSettingsSource(PydanticBaseSettingsSource):
    def get_field_value(self, field, field_name):
        # Fetch from remote config service
        response = httpx.get(f"https://config.example.com/{field_name}")
        return response.json()['value']

Extension 3: Configuration Validation Webhook

Send configuration to a validation service:

config-manager validate --webhook=https://security.example.com/audit

Extension 4: Secret Rotation

Track when secrets were last rotated:

class Settings(BaseSettings):
    db_password: SecretStr
    db_password_rotated_at: datetime

    @model_validator(mode='after')
    def check_secret_age(self):
        age = datetime.now() - self.db_password_rotated_at
        if age > timedelta(days=90):
            warnings.warn("Database password should be rotated")
        return self

Extension 5: Configuration Encryption

Encrypt sensitive .env files at rest:

config-manager encrypt .env --key=master.key
config-manager decrypt .env.encrypted --key=master.key

Real-World Connections

Where This Pattern Appears

  1. Django Settings: settings.py with environment overrides
  2. FastAPI + Pydantic: The standard pattern for FastAPI apps
  3. Kubernetes ConfigMaps: Injected as environment variables
  4. AWS Parameter Store: Configuration management at scale
  5. HashiCorp Vault: Secret management

Industry Examples

  • Django-environ: Environment-based Django configuration
  • python-decouple: Strict separation of settings from code
  • dynaconf: Multi-layer configuration management
  • Pydantic Settings: The modern, type-safe approach

Production Considerations

  1. Never commit secrets: Use .gitignore for .env files
  2. Use secret management: Vault, AWS Secrets Manager, etc.
  3. Validate on startup: Fail fast if configuration is invalid
  4. Log configuration changes: Audit trail for compliance
  5. Use different secrets per environment: Never share between dev/prod

Self-Assessment Checklist

Core Understanding

  • Can I explain the settings source precedence?
  • Can I describe how environment variables map to nested settings?
  • Can I explain why SecretStr is important?
  • Can I describe the difference between env_prefix and env_nested_delimiter?

Implementation Skills

  • Can I create a multi-level settings hierarchy?
  • Can I load configuration from multiple .env files?
  • Can I implement custom settings sources?
  • Can I generate documentation from settings classes?

Production Readiness

  • Can I validate that configuration is appropriate for production?
  • Can I export configuration without exposing secrets?
  • Can I test settings with various environment configurations?
  • Can I handle missing required settings gracefully?

Mastery Indicators

  • Tool handles all edge cases (missing files, invalid values)
  • Secrets are never exposed in any output
  • Configuration is validated comprehensively
  • Documentation is auto-generated and accurate

Resources

Documentation

Books

  • โ€œArchitecture Patterns with Pythonโ€ by Percival & Gregory - Chapter 11 on Configuration
  • โ€œThe Twelve-Factor Appโ€ by Adam Wiggins (online) - Configuration principles