Social Media Data Mining and Analysis Mastery
Goal
After completing this learning path, you will deeply understand how to extract, process, analyze, and visualize social media data at scale. You will master the complete pipeline from API authentication and rate limit management through sentiment analysis, network graph construction, and real-time dashboard creation. You will be able to build production-grade social listening tools, detect bots and fake accounts, identify influencers algorithmically, and present actionable insights through interactive visualizations. Most importantly, you will understand WHY social networks exhibit certain patterns, how information propagates through communities, and how to extract signal from the noise of millions of posts.
Why Social Media Data Mining Matters
The Scale of Social Data
Social media platforms generate an unprecedented volume of human-generated content:
- X (Twitter): 500+ million posts per day
- Reddit: 1.7 billion comments per month across 100,000+ active communities
- Instagram: 2+ billion monthly active users
- TikTok: 1 billion+ daily video views
This data represents the largest corpus of real-time human opinion, behavior, and social interaction ever collected. Organizations that can effectively mine this data gain competitive advantages in:
- Market Research: Understanding customer sentiment before competitors
- Crisis Detection: Identifying PR issues within minutes, not days
- Trend Prediction: Spotting viral content before it peaks
- Political Analysis: Tracking public opinion shifts in real-time
- Academic Research: Studying human behavior at population scale
The Technical Challenge
Traditional Data Analysis Social Media Data Mining
┌─────────────────────────┐ ┌─────────────────────────┐
│ Structured databases │ │ Unstructured text/media │
│ Clean, validated data │ │ Noisy, messy, real-time │
│ Batch processing │ │ Streaming + batch │
│ SQL queries │ │ NLP + Graph + ML │
│ Simple aggregations │ │ Complex entity linking │
│ Static reports │ │ Live dashboards │
└─────────────────────────┘ └─────────────────────────┘
Social media data mining requires a unique combination of skills:
- API Engineering: Authentication, rate limiting, pagination, backoff strategies
- Natural Language Processing: Tokenization, sentiment, topic modeling, entity extraction
- Graph Theory: Network analysis, community detection, influence measurement
- Time Series Analysis: Trend detection, seasonality, anomaly detection
- Machine Learning: Classification (bot detection), clustering (audience segmentation)
- Data Visualization: Interactive dashboards, real-time updates, storytelling
Career Impact
According to industry surveys, data scientists who can work with social media data command 15-25% higher salaries than general data analysts. Key roles include:
- Social Media Analyst ($65,000-$95,000)
- Marketing Data Scientist ($90,000-$140,000)
- Social Listening Engineer ($100,000-$150,000)
- NLP Research Scientist ($120,000-$180,000)
Core Concept Analysis
The Social Media Data Pipeline
┌─────────────────────────────────────────────────────────────────────┐
│ SOCIAL MEDIA DATA PIPELINE │
└─────────────────────────────────────────────────────────────────────┘
┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐
│ COLLECT │───▶│ CLEAN │───▶│ ANALYZE │───▶│VISUALIZE│
└────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘
│ │ │ │
▼ ▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ APIs │ │ Dedupe │ │ Sentiment│ │ Charts │
│ Scraping │ │ Normalize│ │ Topics │ │ Graphs │
│ Streaming│ │ Enrich │ │ Networks │ │Dashboards│
└──────────┘ └──────────┘ └──────────┘ └──────────┘
│ │ │ │
▼ ▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ Rate │ │ Missing │ │ ML Model │ │ Export │
│ Limits │ │ Values │ │ Training │ │ Reports │
│ Auth │ │ Encoding │ │ Inference│ │ Sharing │
└──────────┘ └──────────┘ └──────────┘ └──────────┘
API Architecture Patterns
Modern social media APIs follow consistent patterns that you must understand:
OAuth 2.0 Authentication Flow:
┌────────────┐ ┌────────────┐
│ Your │ 1. Request auth URL │ OAuth │
│ App │──────────────────────────▶│ Server │
└────────────┘ └────────────┘
│ │
│ 2. User redirected │
▼ │
┌────────────┐ │
│ User │ 3. User grants permission │
│ Browser │─────────────────────────────────┤
└────────────┘ │
│ │
│ 4. Redirect with auth code │
▼ │
┌────────────┐ 5. Exchange code for token │
│ Your │─────────────────────────────────┤
│ App │ │
└────────────┘◀────────────────────────────────┘
│ 6. Access token
│
▼
┌────────────┐ 7. API calls with token
│ API │◀─────────────────────────
│ Endpoint │
└────────────┘
Rate Limiting Strategies
Rate Limit Management:
Time Window (15 minutes)
├───────────────────────────────────────────────────────────────┤
│░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░│
└───────────────────────────────────────────────────────────────┘
^ ^ ^ ^
│ │ │ │
100 200 300 450
requests requests requests limit
Strategies:
┌─────────────────────────────────────────────────────────────────┐
│ 1. FIXED WINDOW │
│ - Count requests per window │
│ - Reset counter at window boundary │
│ - Risk: Burst at window edges │
├─────────────────────────────────────────────────────────────────┤
│ 2. SLIDING WINDOW │
│ - Track timestamp of each request │
│ - Count requests in last N minutes │
│ - Smoother distribution │
├─────────────────────────────────────────────────────────────────┤
│ 3. EXPONENTIAL BACKOFF │
│ - On 429 error: wait 1s, retry │
│ - On second 429: wait 2s, retry │
│ - On third 429: wait 4s, retry │
│ - Max backoff: 64s or fail │
├─────────────────────────────────────────────────────────────────┤
│ 4. TOKEN BUCKET │
│ - Bucket holds N tokens │
│ - Each request consumes 1 token │
│ - Tokens regenerate at fixed rate │
│ - Allows controlled bursting │
└─────────────────────────────────────────────────────────────────┘
Sentiment Analysis Architecture
Sentiment Analysis Pipeline:
Input Text: "This product is absolutely amazing! Best purchase ever 😍"
┌─────────────────────────────────────────────────────────────────┐
│ PREPROCESSING │
├─────────────────────────────────────────────────────────────────┤
│ 1. Lowercase → "this product is absolutely amazing..." │
│ 2. Remove URLs → (none in this example) │
│ 3. Handle @mentions → (none in this example) │
│ 4. Emoji → Text → "...best purchase ever [love_eyes]" │
│ 5. Tokenize → ["this", "product", "is", ...] │
│ 6. Remove stopwords → ["product", "absolutely", "amazing", ...] │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ LEXICON-BASED (VADER) │
├─────────────────────────────────────────────────────────────────┤
│ Word │ Valence │ Modifier │
│──────────────│──────────│───────────────────────────────────────│
│ "amazing" │ +3.1 │ "absolutely" intensifies → +3.4 │
│ "best" │ +3.0 │ │
│ [love_eyes] │ +2.5 │ Emoji bonus │
│──────────────│──────────│───────────────────────────────────────│
│ Compound Score: 0.92 (Very Positive) │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ OUTPUT │
├─────────────────────────────────────────────────────────────────┤
│ { │
│ "text": "This product is absolutely amazing!...", │
│ "sentiment": { │
│ "compound": 0.92, │
│ "positive": 0.78, │
│ "neutral": 0.22, │
│ "negative": 0.00 │
│ }, │
│ "label": "POSITIVE" │
│ } │
└─────────────────────────────────────────────────────────────────┘
Social Network Graph Structure
Follower Network (Directed Graph):
┌───────────────────────────────────────────┐
│ SOCIAL NETWORK GRAPH │
└───────────────────────────────────────────┘
┌─────┐
┌───▶│ A │◀───┐
│ └──┬──┘ │
│ │ │
│ ▼ │
┌────┴──┐ ┌─────┐ ┌──┴────┐
│ B │◀│ C │▶│ D │
└───┬───┘ └──┬──┘ └───┬───┘
│ │ │
│ ▼ │
│ ┌─────┐ │
└───▶│ E │◀─────┘
└─────┘
Node Metrics:
┌─────────────────────────────────────────────────────────────────┐
│ Node │ In-Degree │ Out-Degree │ Betweenness │ PageRank │
│──────│───────────│────────────│─────────────│──────────────────│
│ A │ 2 │ 1 │ 0.00 │ 0.25 │
│ B │ 1 │ 1 │ 0.00 │ 0.18 │
│ C │ 0 │ 2 │ 0.00 │ 0.12 │
│ D │ 1 │ 1 │ 0.00 │ 0.18 │
│ E │ 3 │ 0 │ 0.00 │ 0.27 │
└─────────────────────────────────────────────────────────────────┘
Key Insights:
- Node A: High in-degree = Many followers (influential)
- Node C: High out-degree = Active engager
- Node E: Highest PageRank = Central connector
Topic Modeling with LDA
Latent Dirichlet Allocation (LDA):
Documents (Tweets/Posts):
┌─────────────────────────────────────────────────────────────────┐
│ Doc 1: "The new iPhone camera is incredible for photos" │
│ Doc 2: "Android battery life beats iPhone every time" │
│ Doc 3: "Lakers game tonight was amazing basketball action" │
│ Doc 4: "NBA playoffs getting intense with close games" │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ LDA MODEL TRAINING │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Document-Topic Distribution (θ) │
│ ┌───────────────────────────────────────┐ │
│ │ Doc 1: Topic 1: 0.85 Topic 2: 0.15 │ │
│ │ Doc 2: Topic 1: 0.90 Topic 2: 0.10 │ │
│ │ Doc 3: Topic 1: 0.10 Topic 2: 0.90 │ │
│ │ Doc 4: Topic 1: 0.05 Topic 2: 0.95 │ │
│ └───────────────────────────────────────┘ │
│ │
│ Topic-Word Distribution (φ) │
│ ┌───────────────────────────────────────┐ │
│ │ Topic 1: "iPhone" "camera" "Android" │ ← TECH/PHONES │
│ │ "battery" "photos" "new" │ │
│ │ │ │
│ │ Topic 2: "game" "basketball" "NBA" │ ← SPORTS │
│ │ "playoffs" "Lakers" "action" │ │
│ └───────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
Concept Summary Table
| Concept | What You Must Internalize | Why It Matters |
|---|---|---|
| OAuth 2.0 | Token-based authentication flow with scopes and refresh | Every API requires proper auth; mistakes = blocked access |
| Rate Limiting | Request budgets, backoff strategies, queue management | Exceeding limits = banned IP/account |
| Pagination | Cursor-based vs offset-based, handling incomplete pages | Large datasets require proper iteration |
| Text Normalization | Lowercasing, stemming, lemmatization, stop words | Garbage in = garbage out for NLP |
| Sentiment Lexicons | Word-valence mappings, intensity modifiers, negation | Foundation of rule-based sentiment |
| VADER | Social-media-optimized sentiment with emoji support | 10x faster than ML for similar accuracy |
| Directed Graphs | Edges have direction (follower != following) | Social networks are asymmetric |
| Centrality Metrics | Degree, betweenness, closeness, PageRank | Different measures = different “influence” definitions |
| Community Detection | Modularity optimization, Louvain algorithm | Finding natural clusters in networks |
| LDA | Generative model assuming topics = word distributions | Unsupervised topic discovery |
| Coherence Scores | c_v, c_umass for evaluating topic quality | Picking optimal number of topics |
| Time Series Decomposition | Trend, seasonality, residual components | Separating signal from noise |
| Streaming vs Batch | Real-time processing vs historical analysis | Different architectures for different needs |
Deep Dive Reading by Concept
Data Collection & APIs
| Book | Relevant Chapters | Key Concepts |
|---|---|---|
| Mastering Social Media Mining with Python (Bonzanini) | Ch. 1-4 | API setup, data collection patterns |
| Mining the Social Web (Russell) | Ch. 1-3, 9 | Twitter, Facebook, API design |
| Data Wrangling with Python (Kazil & Jarmul) | Ch. 5-7 | Web scraping, API consumption |
Natural Language Processing
| Book | Relevant Chapters | Key Concepts |
|---|---|---|
| Natural Language Processing with Python (Bird et al.) | Ch. 1-3, 6-7 | NLTK, text processing, classification |
| Speech and Language Processing (Jurafsky & Martin) | Ch. 4-6, 20 | NLP fundamentals, sentiment |
| Applied Text Analysis with Python (Bengfort et al.) | Ch. 4-8, 12 | Topic modeling, classification |
Network Analysis
| Book | Relevant Chapters | Key Concepts |
|---|---|---|
| Networks, Crowds, and Markets (Easley & Kleinberg) | Ch. 2-5, 13-16 | Graph theory, influence, cascades |
| Social Network Analysis (Wasserman & Faust) | Ch. 4-8 | Centrality, structure, roles |
| Network Science (Barabasi) | Ch. 2-4, 9 | Scale-free networks, hubs |
Data Visualization
| Book | Relevant Chapters | Key Concepts |
|---|---|---|
| Python Data Science Handbook (VanderPlas) | Ch. 4 | Matplotlib, visualization basics |
| Storytelling with Data (Knaflic) | Ch. 2-6 | Effective charts, narrative |
| Fundamentals of Data Visualization (Wilke) | Ch. 5-12 | Chart types, best practices |
Machine Learning for Social Media
| Book | Relevant Chapters | Key Concepts |
|---|---|---|
| Hands-On Machine Learning (Geron) | Ch. 3-6 | Classification, pipelines |
| Feature Engineering for Machine Learning (Zheng & Casari) | Ch. 3-5 | Text features, embeddings |
Projects
Project 1: X (Twitter) API Data Collector
What You’ll Build: A robust Python application that authenticates with the X API v2, collects tweets based on search queries or user timelines, handles rate limits gracefully, and stores results in both JSON files and a SQLite database for later analysis.
Why It Teaches the Concept: API interaction is the foundation of all social media mining. You cannot analyze data you cannot collect. This project forces you to understand OAuth 2.0 authentication, rate limit headers, pagination cursors, and data persistence patterns that apply to every social platform API.
Core Challenges:
- Implementing OAuth 2.0 Bearer Token authentication
- Parsing rate limit headers and implementing exponential backoff
- Handling cursor-based pagination for large result sets
- Designing a database schema that captures tweet metadata
- Building a resumable collection system that survives interruptions
Key Concepts: API authentication, rate limiting, pagination, data persistence
| Difficulty: Beginner-Intermediate | Time Estimate: 8-12 hours |
Prerequisites: Basic Python, HTTP fundamentals, SQL basics
Real World Outcome
When complete, running your collector will produce output like:
$ python twitter_collector.py --query "Python programming" --max-tweets 1000
[2024-01-15 10:30:15] Authenticating with X API v2...
[2024-01-15 10:30:16] Authentication successful. Bearer token obtained.
[2024-01-15 10:30:16] Starting collection for query: "Python programming"
[2024-01-15 10:30:17] Retrieved 100 tweets (100/1000)
[2024-01-15 10:30:18] Rate limit: 298/300 remaining, resets in 14:42
[2024-01-15 10:30:19] Retrieved 100 tweets (200/1000)
...
[2024-01-15 10:35:22] Rate limit approaching. Sleeping for 10 seconds...
...
[2024-01-15 10:45:33] Collection complete. 1000 tweets saved.
Summary:
- Tweets collected: 1000
- Unique users: 847
- Date range: 2024-01-10 to 2024-01-15
- Database: tweets.db (2.3 MB)
- JSON export: tweets_20240115_103015.json
The SQLite database will have tables for tweets, users, and hashtags with proper foreign key relationships.
The Core Question You’re Answering
How do you reliably extract large volumes of data from rate-limited APIs without getting blocked, losing data, or violating terms of service?
Concepts You Must Understand First
Before starting, verify you can answer these questions:
- What is the difference between OAuth 1.0a and OAuth 2.0? Which does X API v2 use?
- What does a 429 HTTP status code mean and how should you respond?
- What is cursor-based pagination and why is it preferred over offset-based for large datasets?
- How do you prevent duplicate data when resuming an interrupted collection?
Book Reference: Mining the Social Web, Chapter 9 - Twitter Cookbook
Questions to Guide Your Design
Authentication Design:
- Where will you store your API credentials securely?
- How will you handle token expiration and refresh?
- Should you support multiple accounts for higher rate limits?
Rate Limit Design:
- How will you parse the X-Rate-Limit headers?
- What backoff strategy will you implement?
- Should you proactively slow down before hitting limits?
Data Model Design:
- What fields from each tweet are essential vs nice-to-have?
- How will you normalize users, hashtags, and mentions?
- Should you store raw JSON for future schema changes?
Resilience Design:
- How will you checkpoint progress for resumability?
- What happens if the network connection drops mid-request?
- How will you handle API error responses (400, 401, 403, 500)?
Thinking Exercise
Before coding, trace through this scenario manually:
- You start collecting tweets at 10:00 AM
- Your rate limit is 300 requests per 15 minutes
- Each request returns 100 tweets
- You need to collect 50,000 tweets
Calculate:
- How many requests total?
- How many 15-minute windows?
- What is the minimum time to complete?
- Draw a timeline showing request bursts and waiting periods
The Interview Questions They’ll Ask
- “Explain how you would handle a situation where the API returns partial data before timing out.”
- “How would you scale this collector to handle 10 different search queries simultaneously?”
- “What’s the difference between polling an API and using webhooks/streaming? When would you use each?”
- “How would you ensure your collector complies with the API’s terms of service?”
- “Describe how you would implement idempotency to prevent duplicate tweets.”
Hints in Layers
Hint 1 - Starting Point: Begin with the tweepy library or raw requests. Set up OAuth 2.0 Bearer Token auth first.
Hint 2 - Rate Limiting: Parse the x-rate-limit-remaining and x-rate-limit-reset headers. Use time.sleep() when approaching limits.
Hint 3 - Implementation Approach: Use a generator pattern to yield tweets, storing the pagination cursor after each page. Wrap the main loop in try/except to catch network errors.
Hint 4 - Debugging: Add logging with timestamps. Store raw API responses temporarily for debugging. Test with small limits first (10 tweets, not 10,000).
Books That Will Help
| Topic | Book | Chapter |
|---|---|---|
| X API basics | Mining the Social Web | Ch. 9 |
| OAuth 2.0 | API Security in Action | Ch. 5-6 |
| Rate limiting patterns | Designing Data-Intensive Applications | Ch. 4 |
| SQLite with Python | Python Cookbook | Ch. 6 |
Common Pitfalls & Debugging
| Problem | Root Cause | Fix |
|---|---|---|
401 Unauthorized |
Invalid or expired Bearer token | Regenerate token in developer portal |
429 Too Many Requests |
Exceeded rate limit | Implement exponential backoff |
| Duplicate tweets | Not deduplicating on tweet ID | Use INSERT OR IGNORE with tweet ID as primary key |
| Missing older tweets | API only returns recent tweets | Use Academic Research access for historical data |
| Incomplete user data | Need to expand user fields | Add user.fields parameter to API call |
Learning Milestones
Milestone 1: Successfully authenticate and retrieve a single page of tweets (100 tweets).
Milestone 2: Implement pagination and rate limiting to collect 1,000+ tweets without errors.
Milestone 3: Build a resumable system with database storage that can survive interruptions and restart.
Project 2: Reddit API Data Scraper with PRAW
What You’ll Build: A comprehensive Reddit data collection tool using PRAW (Python Reddit API Wrapper) that can scrape posts and comments from subreddits, user profiles, and search results, with support for multiple sorting methods and export to various formats.
Why It Teaches the Concept: Reddit’s API differs significantly from X’s API in authentication (OAuth for scripts vs apps), data structure (hierarchical comments vs flat tweets), and rate limiting. Building a second platform scraper reinforces API patterns while teaching you to adapt to different data models.
Core Challenges:
- Setting up Reddit OAuth credentials (script vs web app types)
- Navigating Reddit’s hierarchical comment structure (trees, not lists)
- Understanding Reddit’s unique rate limiting (OAuth-based quotas)
- Handling deleted/removed content gracefully
- Flattening comment trees for tabular analysis
Key Concepts: PRAW library, comment trees, OAuth script apps, subreddit filtering
| Difficulty: Beginner-Intermediate | Time Estimate: 6-10 hours |
Prerequisites: Completed Project 1, understanding of tree data structures
Real World Outcome
$ python reddit_scraper.py --subreddit datascience --posts 100 --comments
[2024-01-15 11:00:00] Authenticating with Reddit API...
[2024-01-15 11:00:01] Connected as script app: social_media_analyzer
[2024-01-15 11:00:01] Scraping r/datascience (sort: hot, limit: 100)
[2024-01-15 11:00:05] Retrieved 100 posts
[2024-01-15 11:00:05] Fetching comments for 100 posts...
[2024-01-15 11:00:10] Post 1/100: "Best ML course in 2024?" - 47 comments
[2024-01-15 11:00:12] Post 2/100: "Career change at 35?" - 128 comments
...
[2024-01-15 11:05:33] Complete. Saved to r_datascience_20240115.csv
Summary:
- Posts collected: 100
- Total comments: 3,847
- Unique authors: 1,234
- Average comments/post: 38.5
- Most active post: "Career change at 35?" (128 comments)
The Core Question You’re Answering
How do you efficiently extract and flatten hierarchical discussion data while preserving the parent-child relationships that give comments context?
Concepts You Must Understand First
- What is the difference between a Reddit “script” app and a “web” app?
- How does Reddit structure comments as trees? What is a
MoreCommentsobject? - What is the difference between
hot,new,top, andrisingsorts? - How does Reddit’s rate limiting differ from X’s fixed-window approach?
Book Reference: Mastering Social Media Mining with Python, Chapter 6
Questions to Guide Your Design
Data Model Design:
- How will you represent the parent-child comment relationship in a flat CSV?
- Should you store the depth level of each comment?
- What metadata about the subreddit itself should you capture?
Completeness Design:
- How will you handle
MoreCommentsobjects (Reddit’s lazy-loading)? - Should you limit comment depth to prevent infinite recursion?
- How will you handle deleted or removed content?
Thinking Exercise
Draw the tree structure for this comment thread:
Post: "What's the best programming language?"
├── Comment A: "Python for data science"
│ ├── Reply A1: "Agreed, but R is good too"
│ │ └── Reply A1a: "R is outdated"
│ └── Reply A2: "What about Julia?"
├── Comment B: "JavaScript for web"
│ └── Reply B1: "TypeScript is better"
└── Comment C: "It depends on use case"
Now design a CSV schema that preserves this hierarchy. What columns do you need?
The Interview Questions They’ll Ask
- “How would you handle a thread with 10,000+ comments efficiently?”
- “Explain the tradeoffs between breadth-first and depth-first traversal for comment trees.”
- “How would you detect and handle Reddit’s ‘shadow banning’ of content?”
- “What ethical considerations exist when scraping Reddit data for research?”
- “How would you anonymize Reddit data for publication?”
Hints in Layers
Hint 1: Install PRAW with pip install praw. Create credentials at reddit.com/prefs/apps (choose “script” type).
Hint 2: Use submission.comments.replace_more(limit=0) to expand all MoreComments objects. Set limit to control API calls.
Hint 3: Implement a recursive function to flatten comments. Pass parent_id and depth as parameters.
Hint 4: Use submission.comments.list() after replace_more() to get a flat list while preserving parent IDs.
Common Pitfalls & Debugging
| Problem | Root Cause | Fix |
|---|---|---|
prawcore.exceptions.ResponseException: 401 |
Invalid credentials | Verify client_id/secret match app type |
| Infinite loop in comments | Not handling MoreComments |
Use replace_more() before iterating |
| Missing comments | Rate limited or deleted | Add delays, check body == "[deleted]" |
| Unicode errors | Non-ASCII characters in posts | Use encoding='utf-8' when writing files |
Project 3: Google Sheets Data Aggregator
What You’ll Build: An automated pipeline that takes collected social media data, aggregates it by configurable time windows and dimensions, and exports formatted results to Google Sheets for non-technical stakeholders to view and analyze.
Why It Teaches the Concept: Data scientists often need to share insights with marketers, executives, and researchers who prefer spreadsheets over databases. This project teaches you the Google Sheets API, data aggregation patterns, and the critical skill of formatting data for non-technical audiences.
Core Challenges:
- Setting up Google Cloud credentials and Sheets API access
- Designing aggregations that answer business questions
- Formatting sheets with headers, colors, and charts
- Implementing scheduled updates (append vs overwrite)
- Sharing sheets programmatically with stakeholders
Key Concepts: gspread library, Google Cloud authentication, aggregation patterns, stakeholder communication
| Difficulty: Beginner-Intermediate | Time Estimate: 6-8 hours |
Prerequisites: Completed Projects 1-2, pandas basics
Real World Outcome
$ python sheets_aggregator.py --input tweets.db --sheet "Weekly Social Report"
[2024-01-15 09:00:00] Loading data from tweets.db...
[2024-01-15 09:00:02] Loaded 15,847 tweets from last 7 days
[2024-01-15 09:00:03] Authenticating with Google Sheets API...
[2024-01-15 09:00:04] Creating/updating sheet: "Weekly Social Report"
Creating worksheets:
- Daily Summary (7 rows)
- Top Hashtags (20 rows)
- Sentiment Breakdown (3 rows)
- Hourly Activity (168 rows)
[2024-01-15 09:00:10] Formatting cells and adding charts...
[2024-01-15 09:00:15] Sharing with: marketing@company.com, exec@company.com
[2024-01-15 09:00:16] Complete!
Sheet URL: https://docs.google.com/spreadsheets/d/1abc123...
The resulting Google Sheet will have multiple tabs with formatted tables, conditional formatting showing sentiment colors, and embedded charts.
The Core Question You’re Answering
How do you transform raw social media data into digestible insights that non-technical stakeholders can understand and act upon?
Concepts You Must Understand First
- How does Google Cloud service account authentication work?
- What is the difference between a Google Sheet, a Worksheet, and a Cell range?
- How do you design aggregations that answer “so what?” business questions?
- What formatting principles make spreadsheets readable?
Book Reference: Data Wrangling with Python, Chapter 10
Questions to Guide Your Design
Aggregation Design:
- What time granularity do stakeholders need? (hourly, daily, weekly?)
- What dimensions should you group by? (hashtag, user, sentiment?)
- What metrics matter? (volume, engagement, sentiment?)
Sheet Structure Design:
- How many worksheets do you need?
- What should each worksheet’s purpose be?
- How will you name cells for clarity?
The Interview Questions They’ll Ask
- “How would you handle a scenario where the source data changes schema?”
- “What’s your strategy for updating sheets incrementally vs full refresh?”
- “How would you add data validation to prevent manual editing errors?”
- “Describe how you’d add drill-down capability from summary to detail.”
Hints in Layers
Hint 1: Use gspread library with service account JSON credentials. Share the sheet with the service account email.
Hint 2: Use pandas for aggregations (df.groupby().agg()), then convert to lists for gspread.
Hint 3: Use worksheet.format() for conditional formatting. Apply number formats with worksheet.format("B:B", {"numberFormat": {"type": "NUMBER"}}).
Hint 4: Create a template sheet manually first, then replicate the formatting programmatically.
Project 4: Pandas Data Cleaning Pipeline
What You’ll Build: A comprehensive data cleaning module that takes raw social media exports (JSON/CSV) and produces analysis-ready DataFrames by handling missing values, normalizing text, parsing dates, extracting entities, and detecting duplicates.
Why It Teaches the Concept: Raw social media data is notoriously messy. Tweets contain emojis, URLs, mentions, and hashtags mixed with text. Timestamps may be in different formats. User data may be missing or malformed. This project teaches you that 80% of data science is data preparation.
Core Challenges:
- Handling mixed-type columns (strings, numbers, nulls)
- Parsing various timestamp formats to consistent datetime
- Extracting URLs, mentions, and hashtags into separate columns
- Normalizing Unicode and handling emojis
- Deduplicating near-duplicate posts
Key Concepts: pandas, text normalization, regex, datetime parsing, deduplication
| Difficulty: Intermediate | Time Estimate: 8-12 hours |
Prerequisites: pandas basics, regex fundamentals
Real World Outcome
$ python clean_data.py --input raw_tweets.json --output cleaned_tweets.parquet
[2024-01-15 10:00:00] Loading raw data...
[2024-01-15 10:00:02] Loaded 50,000 rows
Data Quality Report (Before):
- Missing values: 12,847 (across all columns)
- Duplicate IDs: 234
- Invalid dates: 89
- Unicode issues: 1,203
Cleaning Steps:
[1/8] Removing duplicates... (234 removed)
[2/8] Parsing timestamps... (89 fixed)
[3/8] Normalizing text... (1,203 fixed)
[4/8] Extracting URLs... (23,456 URLs found)
[5/8] Extracting mentions... (18,234 mentions found)
[6/8] Extracting hashtags... (31,567 hashtags found)
[7/8] Filling missing values... (12,847 handled)
[8/8] Type casting... (complete)
Data Quality Report (After):
- Missing values: 0
- Duplicate IDs: 0
- Invalid dates: 0
- Unicode issues: 0
[2024-01-15 10:00:45] Saved to cleaned_tweets.parquet (49,766 rows)
The Core Question You’re Answering
How do you systematically transform chaotic, real-world data into a consistent format that downstream analysis can trust?
Concepts You Must Understand First
- What is the difference between
None,NaN, and empty string in pandas? - How does Unicode normalization work (NFC, NFD, NFKC, NFKD)?
- What regex patterns match URLs, @mentions, and #hashtags?
- What is the difference between exact and fuzzy deduplication?
Book Reference: Python for Data Analysis (McKinney), Chapters 7-8
Questions to Guide Your Design
Missing Value Strategy:
- Which columns can have missing values filled with defaults?
- Which missing values indicate data quality issues?
- Should you drop rows with critical missing values or flag them?
Text Normalization Strategy:
- Should you lowercase all text? (Consider case-sensitivity of @mentions)
- How will you handle emojis? (Keep, remove, or convert to text?)
- Should you expand contractions? (“don’t” → “do not”)
Thinking Exercise
Given this raw tweet text:
"Just saw @elonmusk's tweet about #Tesla 🚗💨 Check it out: https://t.co/abc123 RT @techcrunch: Amazing!"
Write out exactly what the cleaned and extracted fields should look like:
clean_text: ?urls: ?mentions: ?hashtags: ?has_retweet: ?emoji_text: ?
The Interview Questions They’ll Ask
- “How would you handle a column that’s sometimes a string and sometimes a list?”
- “Explain your strategy for detecting and handling outliers in engagement metrics.”
- “How would you validate that your cleaning pipeline didn’t corrupt the data?”
- “What’s the difference between dropping and imputing missing values? When would you use each?”
Hints in Layers
Hint 1: Start with df.info() and df.describe() to understand your data. Check df.isnull().sum() for missing values.
Hint 2: Use pd.to_datetime(df['created_at'], errors='coerce') to parse dates, then handle NaT values.
Hint 3: Regex patterns: URLs https?://\S+, mentions @\w+, hashtags #\w+.
Hint 4: For near-duplicate detection, consider using fuzzywuzzy or comparing Jaccard similarity of word sets.
Project 5: Sentiment Analyzer with VADER and TextBlob
What You’ll Build: A sentiment analysis pipeline that processes social media text using multiple methods (VADER, TextBlob, and a simple ML classifier), compares their outputs, and produces consensus sentiment scores optimized for social media language.
Why It Teaches the Concept: Sentiment analysis is the most requested social media analysis task. However, no single method is perfect. VADER excels at social media slang and emojis, TextBlob is simpler but more general, and ML classifiers can be trained on domain-specific data. Understanding their tradeoffs is essential.
Core Challenges:
- Preprocessing text specifically for sentiment (different from general cleaning)
- Understanding VADER’s compound score calculation
- Handling negation, intensifiers, and sarcasm
- Calibrating thresholds for positive/negative/neutral labels
- Ensembling multiple methods for robust predictions
Key Concepts: VADER, TextBlob, polarity, subjectivity, ensemble methods
| Difficulty: Intermediate | Time Estimate: 10-14 hours |
Prerequisites: Completed Project 4, basic NLP understanding
Real World Outcome
$ python sentiment_analyzer.py --input cleaned_tweets.parquet --output sentiment_results.csv
[2024-01-15 11:00:00] Loading 49,766 tweets...
[2024-01-15 11:00:01] Preprocessing for sentiment analysis...
Analyzing with multiple methods:
[1/3] VADER analysis... (3.2 seconds)
[2/3] TextBlob analysis... (6.5 seconds)
[3/3] Ensemble consensus... (0.1 seconds)
Method Comparison:
┌──────────────┬──────────┬──────────┬─────────┐
│ Method │ Positive │ Negative │ Neutral │
├──────────────┼──────────┼──────────┼─────────┤
│ VADER │ 45.2% │ 22.1% │ 32.7% │
│ TextBlob │ 38.9% │ 18.3% │ 42.8% │
│ Ensemble │ 42.0% │ 20.2% │ 37.8% │
└──────────────┴──────────┴──────────┴─────────┘
Agreement Rate: 78.3% (all methods agree)
Disagreement Analysis:
- VADER positive, TextBlob neutral: 8.2%
- VADER negative, TextBlob neutral: 5.1%
- Other disagreements: 8.4%
Sample Disagreements (for review):
"This is sick! 🔥" → VADER: +0.75, TextBlob: -0.2 (slang issue)
"Yeah, right..." → VADER: +0.1, TextBlob: +0.0 (sarcasm issue)
[2024-01-15 11:00:12] Saved to sentiment_results.csv
The Core Question You’re Answering
How do you accurately measure the emotional valence of short, informal, emoji-laden social media text when context is limited and language evolves rapidly?
Concepts You Must Understand First
- What is the difference between polarity and subjectivity?
- How does VADER handle “booster words” (very, extremely) and “dampener words” (slightly, kind of)?
- Why does capitalization affect VADER scores but not TextBlob?
- What is the “but” clause rule in sentiment (negating prior sentiment)?
Book Reference: Natural Language Processing with Python, Chapter 6
Questions to Guide Your Design
Preprocessing Decisions:
- Should you remove @mentions before sentiment analysis?
- How should you handle quoted text and retweets?
- Should emojis be kept, removed, or translated to text?
Threshold Decisions:
- At what compound score does “neutral” become “positive”?
- Should thresholds be symmetric (0.05) or asymmetric (positive: 0.05, negative: -0.1)?
Ensemble Design:
- How will you weight the different methods?
- What do you do when methods disagree?
The Interview Questions They’ll Ask
- “VADER gives ‘This movie is not bad’ a positive score. Is this correct? How would you handle negation?”
- “How would you detect sarcasm in social media text?”
- “Your client says your sentiment scores don’t match their intuition. How do you investigate?”
- “How would you build a domain-specific sentiment lexicon for financial tweets?”
- “What are the limitations of lexicon-based sentiment analysis?”
Hints in Layers
Hint 1: Install with pip install vaderSentiment textblob. VADER returns {'pos': 0.5, 'neg': 0.0, 'neu': 0.5, 'compound': 0.6}.
Hint 2: VADER’s compound score is normalized between -1 and +1. Recommended thresholds: positive >= 0.05, negative <= -0.05.
Hint 3: For ensemble, try weighted voting: final = 0.6 * vader + 0.4 * textblob (VADER weighted higher for social media).
Hint 4: Create a test set of 100 manually labeled tweets to validate your approach. Track precision/recall by sentiment class.
Project 6: Trending Topics Detector
What You’ll Build: A real-time trending topics detection system that identifies suddenly popular hashtags, keywords, and phrases by comparing current frequency to historical baselines, with burst detection algorithms.
Why It Teaches the Concept: Trending detection is not just “what’s popular”—it’s “what’s MORE popular than expected.” A hashtag with 1,000 tweets might not be trending if it normally has 900. But a hashtag with 100 tweets is trending if it normally has 10. This project teaches statistical anomaly detection.
Core Challenges:
- Building rolling historical baselines per hashtag
- Implementing Z-score anomaly detection
- Handling new hashtags with no history
- Distinguishing organic trends from coordinated campaigns
- Ranking trends by velocity (acceleration of growth)
Key Concepts: Anomaly detection, Z-scores, time series baselines, burst detection
| Difficulty: Intermediate-Advanced | Time Estimate: 12-16 hours |
Prerequisites: Statistics basics (mean, standard deviation), time series fundamentals
Real World Outcome
$ python trend_detector.py --window 1h --baseline 7d
[2024-01-15 12:00:00] Loading historical data (last 7 days)...
[2024-01-15 12:00:05] Building per-hashtag baselines...
[2024-01-15 12:00:10] Analyzing current window (last 1 hour)...
TRENDING NOW (sorted by Z-score):
┌─────┬─────────────────────┬─────────┬──────────┬─────────┬────────────┐
│ Rank│ Hashtag │ Current │ Expected │ Z-Score │ Velocity │
├─────┼─────────────────────┼─────────┼──────────┼─────────┼────────────┤
│ 1 │ #SuperBowlAds │ 2,847 │ 45 │ +42.3 │ ↑↑↑ 6228% │
│ 2 │ #TaylorSwift │ 1,234 │ 312 │ +12.1 │ ↑↑ 296% │
│ 3 │ #CryptoNews │ 892 │ 234 │ +8.7 │ ↑ 281% │
│ 4 │ #TechLayoffs │ 567 │ 123 │ +6.2 │ ↑ 361% │
│ 5 │ #MondayMotivation │ 456 │ 234 │ +2.1 │ ↗ 95% │
└─────┴─────────────────────┴─────────┴──────────┴─────────┴────────────┘
NEW TRENDS (no historical baseline):
#ProductLaunch2024 - 234 mentions (monitoring)
#BreakingNews123 - 156 mentions (monitoring)
SUSPICIOUS PATTERNS:
⚠️ #BuyNowCrypto - 98% of tweets from accounts < 30 days old
The Core Question You’re Answering
How do you distinguish genuinely viral content from noise, scheduled events, and coordinated inauthentic behavior using statistical methods?
Concepts You Must Understand First
- What is a Z-score and how do you interpret values of 1, 2, and 3?
- How does exponential moving average (EMA) differ from simple moving average?
- What is the “cold start” problem for new hashtags?
- What signals indicate coordinated (potentially fake) trending behavior?
Book Reference: Time Series Analysis (Box-Jenkins), Chapter 2
Thinking Exercise
Given this hashtag data:
| Hour | #CoffeeLovers | #SuperBowl |
|---|---|---|
| H-6 | 100 | 50 |
| H-5 | 95 | 55 |
| H-4 | 105 | 48 |
| H-3 | 98 | 52 |
| H-2 | 102 | 60 |
| H-1 | 110 | 500 |
| NOW | 115 | 5000 |
Calculate the Z-score for each hashtag’s current count. Which is “more trending”?
The Interview Questions They’ll Ask
- “How would you handle hashtags that trend at the same time every week (e.g., #MondayMotivation)?”
- “What’s the difference between ‘trending’ and ‘popular’?”
- “How would you detect coordinated campaigns trying to artificially trend a hashtag?”
- “How would you balance sensitivity (catching real trends) vs specificity (avoiding false positives)?”
Project 7: Follower Network Analyzer with NetworkX
What You’ll Build: A social network graph analysis tool that constructs follower/following networks from collected user data, calculates centrality metrics, detects communities, and visualizes the network structure.
Why It Teaches the Concept: Social media is fundamentally about connections. Understanding who follows whom, who influences whom, and how communities form requires graph theory. This project teaches you to see social data not as rows in a table but as nodes and edges in a network.
Core Challenges:
- Constructing directed graphs from follower lists (API rate limits!)
- Computing centrality metrics at scale (degree, betweenness, PageRank)
- Detecting communities using modularity optimization
- Handling very large graphs (100K+ nodes)
- Creating meaningful visualizations of network structure
Key Concepts: NetworkX, directed graphs, centrality, community detection, Louvain algorithm
| Difficulty: Intermediate-Advanced | Time Estimate: 14-18 hours |
Prerequisites: Graph theory basics, completed Projects 1-2
Real World Outcome
$ python network_analyzer.py --seed-users "@techleader,@mlexpert" --depth 2
[2024-01-15 13:00:00] Starting network crawl from 2 seed users...
[2024-01-15 13:00:01] Depth 0: 2 users
[2024-01-15 13:05:00] Depth 1: 847 users (followers of seeds)
[2024-01-15 13:45:00] Depth 2: 12,456 users (sample of depth-1 followers)
Network Statistics:
┌────────────────────────────────────────┐
│ Nodes (users): 13,305 │
│ Edges (follow relationships): 45,678 │
│ Density: 0.00026 │
│ Average degree: 3.43 │
│ Diameter: 8 │
│ Clustering coefficient: 0.34 │
└────────────────────────────────────────┘
Top Users by Centrality:
┌─────────────────┬──────────┬─────────────┬──────────┐
│ User │ PageRank │ Betweenness │ In-Degree│
├─────────────────┼──────────┼─────────────┼──────────┤
│ @techleader │ 0.0234 │ 0.0567 │ 1,234 │
│ @mlexpert │ 0.0198 │ 0.0345 │ 987 │
│ @datasciencemag │ 0.0156 │ 0.0234 │ 756 │
│ @airesearcher │ 0.0134 │ 0.0189 │ 654 │
│ @pythondev │ 0.0098 │ 0.0123 │ 543 │
└─────────────────┴──────────┴─────────────┴──────────┘
Communities Detected: 8
Community 1: "ML Research" (2,345 users) - Top: @mlexpert, @airesearcher
Community 2: "Tech Startup" (1,987 users) - Top: @techleader, @vcfunder
Community 3: "Data Engineering" (1,654 users) - Top: @datasciencemag
...
[2024-01-15 13:50:00] Saved network to network.graphml
[2024-01-15 13:50:01] Saved visualization to network_viz.html
The Core Question You’re Answering
How do you identify the most influential users in a network, and what different definitions of “influence” exist mathematically?
Concepts You Must Understand First
- What is the difference between an undirected and directed graph?
- Why is “in-degree” more meaningful than “out-degree” for influence?
- How does PageRank differ from simple follower count?
- What does “betweenness centrality” measure?
Book Reference: Networks, Crowds, and Markets, Chapters 2-4
Questions to Guide Your Design
Graph Construction:
- Should you include follow relationships in both directions?
- How will you handle users with millions of followers (API limits)?
- Should you weight edges by interaction frequency?
Scalability:
- At what graph size does in-memory NetworkX become impractical?
- How will you sample large networks?
- Should you use approximate algorithms for centrality?
The Interview Questions They’ll Ask
- “Explain the difference between degree centrality, betweenness centrality, and eigenvector centrality.”
- “How would you identify ‘bridge’ users who connect otherwise separate communities?”
- “Your client wants to find influencers for a marketing campaign. What metrics would you use?”
- “How would you detect fake followers in a network graph?”
- “What does a high clustering coefficient tell you about a network?”
Hints in Layers
Hint 1: Use networkx.DiGraph() for directed follower networks. Add edges with G.add_edge(follower, followed).
Hint 2: For PageRank: nx.pagerank(G). For communities: pip install python-louvain then community_louvain.best_partition(G.to_undirected()).
Hint 3: For visualization of large graphs, use pyvis or export to Gephi format with nx.write_gexf(G, 'network.gexf').
Hint 4: Sample networks by taking only the top-N followers by follower count to avoid API exhaustion.
Project 8: Hashtag Co-occurrence Analyzer
What You’ll Build: A hashtag analysis system that identifies which hashtags appear together, calculates co-occurrence strength using various metrics (Jaccard, PMI, lift), and visualizes hashtag communities to understand topic clustering.
Why It Teaches the Concept: Hashtags are the primary way users categorize their content. Understanding which hashtags cluster together reveals topic structures, audience overlap, and content strategies. This is a form of market basket analysis applied to social media.
Core Challenges:
- Extracting and normalizing hashtags from text
- Building co-occurrence matrices efficiently
- Calculating meaningful similarity metrics
- Filtering noise (low-frequency, generic hashtags)
- Visualizing hashtag networks
Key Concepts: Co-occurrence matrices, Jaccard similarity, PMI, association rules
| Difficulty: Intermediate | Time Estimate: 8-12 hours |
Prerequisites: Basic linear algebra (matrices), completed Project 4
Real World Outcome
$ python hashtag_analyzer.py --input cleaned_tweets.parquet --min-count 50
[2024-01-15 14:00:00] Extracting hashtags from 49,766 tweets...
[2024-01-15 14:00:02] Found 3,456 unique hashtags
[2024-01-15 14:00:02] Filtering to hashtags with >= 50 occurrences: 234
Building co-occurrence matrix (234 x 234)...
[2024-01-15 14:00:05] Complete. 12,456 non-zero pairs.
Top Hashtag Pairs by Lift:
┌─────────────────────────────┬────────┬─────────┬───────┐
│ Hashtag Pair │ Count │ Jaccard │ Lift │
├─────────────────────────────┼────────┼─────────┼───────┤
│ #Python + #MachineLearning │ 1,234 │ 0.45 │ 8.7 │
│ #DataScience + #AI │ 987 │ 0.38 │ 6.2 │
│ #React + #JavaScript │ 765 │ 0.52 │ 5.8 │
│ #AWS + #DevOps │ 654 │ 0.34 │ 5.1 │
│ #Startup + #Entrepreneur │ 543 │ 0.29 │ 4.3 │
└─────────────────────────────┴────────┴─────────┴───────┘
Hashtag Communities:
Cluster 1: #Python, #MachineLearning, #DataScience, #AI, #DeepLearning
Cluster 2: #JavaScript, #React, #NodeJS, #WebDev, #Frontend
Cluster 3: #AWS, #DevOps, #Cloud, #Docker, #Kubernetes
Cluster 4: #Startup, #Entrepreneur, #VC, #Funding, #Business
[2024-01-15 14:00:10] Saved network to hashtag_network.html
The Core Question You’re Answering
How do you discover latent topic structures in social media by analyzing which labels (hashtags) users apply together?
Concepts You Must Understand First
- What is a co-occurrence matrix and how is it different from a correlation matrix?
- What does Jaccard similarity measure? When is it better than cosine similarity?
- What is PMI (Pointwise Mutual Information) and why does it handle rare items better?
- What is “lift” in association rule mining?
Thinking Exercise
Given these tweets:
- Tweet 1: #Python #DataScience #ML
- Tweet 2: #Python #MachineLearning #AI
- Tweet 3: #Python #WebDev #Django
- Tweet 4: #JavaScript #WebDev #React
- Tweet 5: #DataScience #ML #AI
Construct the co-occurrence matrix for all hashtag pairs. Then calculate Jaccard similarity for #Python/#DataScience and #Python/#WebDev.
Project 9: Engagement Metrics Calculator
What You’ll Build: An engagement analytics engine that calculates and tracks various engagement metrics (engagement rate, virality score, amplification ratio) for posts and accounts, with benchmarking against category averages.
Why It Teaches the Concept: Engagement metrics are what clients and stakeholders actually care about. Raw counts (likes, retweets) are meaningless without context. This project teaches you to derive meaningful, normalized metrics that allow comparison across accounts of different sizes.
Core Challenges:
- Defining engagement rate formulas appropriate for each platform
- Normalizing by follower count without dividing by zero
- Calculating time-decay weighted engagement
- Benchmarking against industry/category averages
- Identifying engagement anomalies (viral posts, bot activity)
Key Concepts: Engagement rate, virality coefficient, amplification, benchmarking
| Difficulty: Intermediate | Time Estimate: 8-12 hours |
Prerequisites: Statistics basics, completed Projects 1-4
Real World Outcome
$ python engagement_analyzer.py --user @techcompany --period 30d
[2024-01-15 15:00:00] Fetching data for @techcompany (last 30 days)...
[2024-01-15 15:00:10] Analyzing 245 posts...
Account Overview:
Followers: 125,000
Posts analyzed: 245
Total impressions: 4,567,890
Engagement Metrics:
┌─────────────────────────┬───────────┬────────────┬──────────┐
│ Metric │ Value │ Benchmark │ vs Avg │
├─────────────────────────┼───────────┼────────────┼──────────┤
│ Engagement Rate │ 3.2% │ 1.8% │ +78% │
│ Amplification Ratio │ 2.4:1 │ 1.5:1 │ +60% │
│ Conversation Rate │ 0.8% │ 0.3% │ +167% │
│ Virality Score │ 12.3 │ 5.0 │ +146% │
└─────────────────────────┴───────────┴────────────┴──────────┘
Top Performing Posts:
┌─────┬────────────────────────────────────┬────────┬────────┐
│ Rank│ Post Preview │ Eng.% │ Shares │
├─────┼────────────────────────────────────┼────────┼────────┤
│ 1 │ "Announcing our new product..." │ 12.3% │ 4,567 │
│ 2 │ "Behind the scenes at..." │ 8.7% │ 2,345 │
│ 3 │ "Tips for developers..." │ 6.5% │ 1,234 │
└─────┴────────────────────────────────────┴────────┴────────┘
Engagement by Content Type:
Videos: 5.2% avg (highest)
Images: 3.1% avg
Text only: 1.8% avg (lowest)
Best Posting Times:
Tuesday 10:00 AM: 4.1% avg engagement
Thursday 2:00 PM: 3.8% avg engagement
The Core Question You’re Answering
How do you measure the quality of social media content in a way that is comparable across accounts, platforms, and time periods?
Concepts You Must Understand First
- Why is engagement RATE more meaningful than raw engagement COUNT?
- What is the difference between “reach” and “impressions”?
- How do you calculate amplification ratio (shares / total engagements)?
- What benchmarks are typical for different industries?
The Interview Questions They’ll Ask
- “An account has 1M followers but only 0.1% engagement rate. Another has 10K followers with 5% engagement. Which is more ‘successful’?”
- “How would you detect when an account is buying fake engagement?”
- “How do you account for the fact that engagement rates naturally decrease as follower count increases?”
- “What’s the difference between engagement rate calculated by reach vs by followers?”
Project 10: Data Visualization Dashboard with Matplotlib/Seaborn
What You’ll Build: A comprehensive visualization module that generates publication-quality charts for social media analytics, including time series, heatmaps, network graphs, word clouds, and comparative bar charts, with consistent styling.
Why It Teaches the Concept: Data without visualization is just numbers. The ability to create clear, compelling visualizations that tell a story is what separates data scientists from data processors. This project teaches you to match chart types to data types and insights.
Core Challenges:
- Selecting appropriate chart types for different data
- Creating consistent visual styling across all charts
- Handling different data scales (log vs linear)
- Adding annotations that highlight key insights
- Exporting in multiple formats (PNG, SVG, PDF)
Key Concepts: Matplotlib, Seaborn, chart selection, visual hierarchy, annotations
| Difficulty: Intermediate | Time Estimate: 10-14 hours |
Prerequisites: Matplotlib basics, completed Projects 4-9
Real World Outcome
Running the visualization module produces a set of styled charts:
$ python visualizer.py --input analysis_results.json --output charts/
[2024-01-15 16:00:00] Loading analysis results...
[2024-01-15 16:00:01] Generating visualizations...
Charts created:
[1/8] Tweet volume time series → charts/volume_timeseries.png
[2/8] Sentiment distribution → charts/sentiment_dist.png
[3/8] Engagement heatmap (day x hour) → charts/engagement_heatmap.png
[4/8] Top hashtags bar chart → charts/top_hashtags.png
[5/8] Influencer network graph → charts/network.png
[6/8] Word cloud by sentiment → charts/wordcloud.png
[7/8] Topic distribution → charts/topics.png
[8/8] Comparative metrics → charts/benchmarks.png
[2024-01-15 16:00:15] All charts saved with consistent styling.
Each chart follows a consistent color palette, typography, and layout that could be used directly in a professional report.
The Core Question You’re Answering
How do you choose the right visualization for each type of insight, and how do you create charts that communicate clearly without explanation?
Questions to Guide Your Design
Chart Selection:
- Time series data → Line charts or area charts
- Distributions → Histograms, box plots, violin plots
- Comparisons → Bar charts, dot plots
- Correlations → Scatter plots, heatmaps
- Parts of whole → Pie charts (sparingly), stacked bars
- Networks → Force-directed graphs, adjacency matrices
Styling Decisions:
- What color palette conveys your brand/message?
- How do you handle colorblind accessibility?
- What font size ensures readability at different scales?
Project 11: Jupyter Notebook Report Generator
What You’ll Build: An automated report generation system that combines data analysis, visualizations, and narrative text into reproducible Jupyter notebooks and exports them to HTML/PDF for stakeholder distribution.
Why It Teaches the Concept: The best analysis is useless if it cannot be communicated. This project teaches you to create reproducible, shareable analysis documents that combine code, results, and interpretation in a single package.
Core Challenges:
- Structuring notebooks for readability (not just execution)
- Parameterizing notebooks for different date ranges/queries
- Adding markdown narrative between code cells
- Exporting to stakeholder-friendly formats
- Scheduling automated report generation
Key Concepts: Jupyter, nbconvert, papermill, narrative structure
| Difficulty: Intermediate | Time Estimate: 8-12 hours |
Prerequisites: Jupyter basics, completed Projects 4-10
Real World Outcome
$ python generate_report.py --template weekly_report.ipynb --start 2024-01-08 --end 2024-01-15
[2024-01-15 17:00:00] Loading template notebook...
[2024-01-15 17:00:01] Injecting parameters: start=2024-01-08, end=2024-01-15
[2024-01-15 17:00:01] Executing notebook...
- Cell 1/15: Loading data... ✓
- Cell 2/15: Computing metrics... ✓
- Cell 3/15: Generating charts... ✓
...
- Cell 15/15: Summary... ✓
[2024-01-15 17:02:30] Notebook executed successfully.
[2024-01-15 17:02:31] Converting to HTML...
[2024-01-15 17:02:35] Converting to PDF...
Output:
- reports/weekly_2024-01-15.ipynb (executable notebook)
- reports/weekly_2024-01-15.html (web viewable)
- reports/weekly_2024-01-15.pdf (printable)
The Core Question You’re Answering
How do you create analysis reports that are both reproducible (for you) and consumable (for stakeholders)?
Project 12: Time Series Analysis of Post Volume
What You’ll Build: A time series analysis module that decomposes social media posting patterns into trend, seasonality, and residual components, forecasts future activity, and detects anomalous volume spikes or drops.
Why It Teaches the Concept: Social media activity has strong temporal patterns—daily cycles, weekly patterns, seasonal trends. Understanding these patterns helps you set appropriate baselines, detect anomalies, and forecast capacity needs.
Core Challenges:
- Resampling data to consistent time intervals
- Decomposing series into trend + seasonality + residual
- Handling missing time periods (nights, weekends)
- Forecasting with ARIMA or Prophet
- Detecting anomalies using statistical thresholds
Key Concepts: Time series decomposition, STL, ARIMA, Prophet, anomaly detection
| Difficulty: Intermediate-Advanced | Time Estimate: 12-16 hours |
Prerequisites: Statistics, pandas datetime handling
Real World Outcome
$ python timeseries_analyzer.py --input tweets.db --granularity hourly
[2024-01-15 18:00:00] Loading time series data...
[2024-01-15 18:00:02] 720 hourly observations (30 days)
Time Series Decomposition:
┌─────────────────────────────────────────────────────────────────┐
│ Component │ Description │
├──────────────┼──────────────────────────────────────────────────│
│ Trend │ Slight upward trend (+2.3% over period) │
│ Weekly │ Higher on weekdays, lowest Sunday │
│ Daily │ Peak at 10AM & 2PM EST, lowest 3-6AM │
│ Residual │ 3 anomalies detected │
└─────────────────────────────────────────────────────────────────┘
Detected Anomalies:
⚠️ 2024-01-10 14:00: 3.2x expected volume (product announcement)
⚠️ 2024-01-12 02:00: 0.1x expected volume (API outage)
⚠️ 2024-01-14 16:00: 2.8x expected volume (trending topic)
7-Day Forecast:
Expected total volume: 12,450 ± 1,200 tweets
Peak day: Tuesday (2,100 tweets)
Low day: Sunday (1,450 tweets)
The Core Question You’re Answering
How do you separate predictable patterns (when people naturally post more) from true signals (something unusual happened)?
Concepts You Must Understand First
- What is stationarity and why does it matter for time series models?
- What is the difference between additive and multiplicative decomposition?
- What are ACF and PACF plots used for?
- How does Prophet handle holidays and special events?
The Interview Questions They’ll Ask
- “How would you differentiate between a viral event and normal daily variation?”
- “Your time series has missing data for 3 days. How do you handle this?”
- “Explain the difference between ARIMA and Prophet. When would you use each?”
- “How would you forecast posting volume during an upcoming product launch?”
Project 13: Bot Detection Classifier
What You’ll Build: A machine learning classifier that identifies likely bot accounts based on behavioral signals (posting frequency, timing patterns, content similarity, network features) with explainable predictions.
Why It Teaches the Concept: Bots pollute social media data. Before any analysis, you must identify and filter (or flag) inauthentic accounts. This project teaches you to think about what makes human behavior different from automated behavior.
Core Challenges:
- Defining features that distinguish bots from humans
- Obtaining labeled training data (known bots/humans)
- Handling class imbalance (few known bots)
- Explaining predictions (why is this account flagged?)
- Keeping up with evolving bot sophistication
Key Concepts: Feature engineering, classification, precision/recall tradeoffs, explainability
| Difficulty: Advanced | Time Estimate: 16-20 hours |
Prerequisites: Machine learning basics, completed Projects 1-7
Real World Outcome
$ python bot_detector.py --input users.csv --model models/bot_classifier.pkl
[2024-01-15 19:00:00] Loading 10,000 users...
[2024-01-15 19:00:05] Extracting features (47 features per user)...
[2024-01-15 19:00:30] Running classification...
Results:
┌──────────────────────────────────────────────────────────────┐
│ Total users analyzed: 10,000 │
│ Likely bots: 847 (8.5%) │
│ Likely humans: 8,923 (89.2%) │
│ Uncertain: 230 (2.3%) │
└──────────────────────────────────────────────────────────────┘
Bot Confidence Distribution:
High confidence (>90%): 234 accounts
Medium confidence (70-90%): 413 accounts
Low confidence (50-70%): 200 accounts
Top Bot Indicators (by feature importance):
1. Tweets per day: avg 89.3 (humans: 4.2)
2. Default profile picture: 78% (humans: 5%)
3. Account age: avg 23 days (humans: 847 days)
4. Posting time entropy: 0.12 (humans: 0.89)
5. Unique content ratio: 0.23 (humans: 0.94)
Sample Flagged Accounts:
┌─────────────────┬────────┬────────────────────────────────────────┐
│ Account │ Score │ Top Reasons │
├─────────────────┼────────┼────────────────────────────────────────┤
│ @user8374823 │ 98.2% │ 200 tweets/day, default avatar, new │
│ @promo_deal123 │ 94.5% │ 100% promotional, same time daily │
│ @news_bot_xyz │ 91.2% │ Automated patterns, low engagement │
└─────────────────┴────────┴────────────────────────────────────────┘
The Core Question You’re Answering
What behavioral patterns distinguish automated social media accounts from genuine human users?
Concepts You Must Understand First
- What features might indicate bot behavior? (timing, content, network, metadata)
- Why is precision vs recall particularly important for bot detection?
- What is the cost of false positives (flagging humans) vs false negatives (missing bots)?
- How do sophisticated bots evade detection?
Questions to Guide Your Design
Feature Engineering:
- Timing features: Do they post at inhuman hours? Perfect intervals?
- Content features: Duplicate posts? Templates? Spam patterns?
- Network features: Following/follower ratio? Engagement rate?
- Metadata features: Default avatar? Random username? Account age?
Model Selection:
- Should you use interpretable models (Random Forest) or black-box (Neural Network)?
- How will you handle class imbalance?
- What threshold will you use for flagging?
The Interview Questions They’ll Ask
- “A marketing account posts 50 times per day on a schedule. Is it a bot?”
- “How would you build a labeled dataset of known bots for training?”
- “Your model has 95% accuracy but only 20% recall on bots. What’s wrong?”
- “How would you detect a new type of bot that evades your current features?”
- “What ethical considerations exist when labeling accounts as bots?”
Project 14: Influencer Identification Algorithm
What You’ll Build: An influencer scoring and ranking system that combines multiple signals (reach, engagement, authority, relevance) into a composite influence score, with the ability to find influencers in specific topic niches.
Why It Teaches the Concept: “Influencer” is a vague term. Someone with 1M followers but 0.1% engagement is not influential. Someone with 10K highly engaged followers in a specific niche may be very influential for that topic. This project teaches you to operationalize abstract concepts.
Core Challenges:
- Defining influence mathematically (there are many valid definitions)
- Combining heterogeneous signals (followers, engagement, network position)
- Normalizing across different account sizes
- Filtering for topic relevance (not just general influence)
- Avoiding gaming and manipulation
Key Concepts: Multi-criteria ranking, normalization, PageRank, topic modeling
| Difficulty: Advanced | Time Estimate: 14-18 hours |
Prerequisites: Completed Projects 7-9, basic ML
Real World Outcome
$ python influencer_finder.py --topic "machine learning" --count 20
[2024-01-15 20:00:00] Loading network data...
[2024-01-15 20:00:10] Filtering for topic relevance: "machine learning"
[2024-01-15 20:00:15] Computing influence scores...
Top 20 Influencers for "Machine Learning":
┌─────┬─────────────────┬────────────┬────────┬────────┬───────────┬────────┐
│Rank │ Account │ Followers │ Eng.% │PageRank│ Relevance │ Score │
├─────┼─────────────────┼────────────┼────────┼────────┼───────────┼────────┤
│ 1 │ @AndrewYNg │ 4.2M │ 2.1% │ 0.0234 │ 0.95 │ 94.2 │
│ 2 │ @ylecun │ 500K │ 3.4% │ 0.0198 │ 0.98 │ 89.5 │
│ 3 │ @kaborovic │ 280K │ 4.2% │ 0.0156 │ 0.89 │ 82.3 │
│ 4 │ @hardmaru │ 210K │ 5.1% │ 0.0145 │ 0.92 │ 78.9 │
│ 5 │ @fchollet │ 180K │ 4.8% │ 0.0134 │ 0.94 │ 76.2 │
│ ... │ ... │ ... │ ... │ ... │ ... │ ... │
└─────┴─────────────────┴────────────┴────────┴────────┴───────────┴────────┘
Influence Score Formula:
Score = 0.25 * log(followers) + 0.30 * engagement + 0.25 * pagerank + 0.20 * relevance
Micro-Influencers (1K-50K followers, high engagement):
@ml_practitioner (12K followers, 8.2% engagement, 0.91 relevance)
@deeplearning_dev (8K followers, 9.1% engagement, 0.88 relevance)
...
The Core Question You’re Answering
How do you quantify “influence” when it’s a multi-dimensional concept that includes reach, engagement, authority, and topical relevance?
Concepts You Must Understand First
- Why is log-transformation appropriate for follower counts?
- How does PageRank measure influence differently than follower count?
- What is topic relevance and how do you calculate it?
- Why might micro-influencers have more impact than mega-influencers for certain campaigns?
The Interview Questions They’ll Ask
- “How would you define ‘influence’ for a B2B vs B2C marketing campaign?”
- “An account has bought followers. How would your algorithm detect this?”
- “How would you find influencers who haven’t been discovered yet (emerging influencers)?”
- “Explain your weighting scheme. Why those weights?”
Project 15: Topic Modeling with LDA
What You’ll Build: A topic modeling pipeline using Latent Dirichlet Allocation that discovers latent topics in social media corpora, assigns topic distributions to documents, and tracks topic evolution over time.
Why It Teaches the Concept: Social media contains millions of posts. You cannot read them all. Topic modeling automatically discovers what people are talking about, enabling you to summarize large corpora and track how conversations evolve.
Core Challenges:
- Preprocessing text for topic modeling (more aggressive than sentiment)
- Selecting the optimal number of topics
- Interpreting and labeling discovered topics
- Handling short documents (tweets are challenging for LDA)
- Tracking topic evolution over time
Key Concepts: LDA, coherence scores, Gensim, topic visualization, pyLDAvis
| Difficulty: Advanced | Time Estimate: 14-18 hours |
Prerequisites: NLP basics, linear algebra fundamentals
Real World Outcome
$ python topic_modeler.py --input cleaned_tweets.parquet --topics 10
[2024-01-15 21:00:00] Loading and preprocessing 49,766 documents...
[2024-01-15 21:00:30] Building dictionary and corpus...
[2024-01-15 21:00:35] Training LDA model (10 topics, 50 passes)...
[2024-01-15 21:05:00] Training complete.
Topic Coherence Analysis:
┌────────────────────────────────────────────────────────────────┐
│ Topics │ Coherence (c_v) │ Recommendation │
├────────┼─────────────────┼─────────────────────────────────────┤
│ 5 │ 0.42 │ Too few - topics overlap │
│ 8 │ 0.51 │ Better separation │
│ 10 │ 0.58 │ Optimal for this corpus ★ │
│ 12 │ 0.55 │ Some topics too similar │
│ 15 │ 0.48 │ Too many - topics fragmented │
└────────┴─────────────────┴─────────────────────────────────────┘
Discovered Topics:
┌───────┬─────────────────────────────────────────────────────────┐
│ Topic │ Top Words (weighted) │
├───────┼─────────────────────────────────────────────────────────┤
│ 1 │ python, data, science, learning, machine (Data Science) │
│ 2 │ job, hiring, remote, salary, career (Job Market) │
│ 3 │ model, training, gpu, tensorflow (Deep Learning) │
│ 4 │ startup, funding, vc, founder (Startups) │
│ 5 │ crypto, bitcoin, blockchain, nft (Crypto/Web3) │
│ 6 │ react, javascript, frontend, css (Web Dev) │
│ 7 │ aws, cloud, devops, kubernetes (Cloud/DevOps) │
│ 8 │ api, rest, graphql, backend (APIs) │
│ 9 │ ai, chatgpt, gpt, openai (Generative AI) │
│ 10 │ security, hack, vulnerability, patch (Security) │
└───────┴─────────────────────────────────────────────────────────┘
Topic Distribution per Day (heatmap saved to topics_over_time.png)
[2024-01-15 21:05:30] Interactive visualization: lda_viz.html
The Core Question You’re Answering
How do you discover what people are talking about when you have too many documents to read manually?
Concepts You Must Understand First
- What generative story does LDA assume about how documents are created?
- What is a “topic” in LDA? (A probability distribution over words)
- How do you choose the number of topics? (Coherence scores, perplexity)
- Why is aggressive preprocessing important for LDA?
Book Reference: Applied Text Analysis with Python, Chapter 8
Thinking Exercise
Given a tweet: “Just deployed my ML model to AWS using Docker. The performance is amazing!”
If your LDA model has topics for “Machine Learning”, “Cloud Computing”, and “Software Development”, what topic distribution would you expect for this tweet?
The Interview Questions They’ll Ask
- “LDA doesn’t work well on tweets. Why? What alternatives exist?”
- “How would you handle a new topic that emerges after your model was trained?”
- “Your stakeholder doesn’t like topic #7’s words. Can you change them?”
- “How would you track how a topic’s prevalence changes over a crisis?”
- “What’s the difference between LDA and BERTopic?”
Project 16: Geographic Analysis and Mapping
What You’ll Build: A geospatial analysis module that extracts location information from social media posts (GPS coordinates, user locations, place mentions), maps activity geographically, and identifies regional patterns in content and sentiment.
Why It Teaches the Concept: Location adds crucial context to social media analysis. “This product sucks” is different if it’s from your target market vs a region you don’t serve. Understanding geographic patterns helps with market analysis, crisis response, and campaign targeting.
Core Challenges:
- Extracting location from multiple sources (GPS, profile, text)
- Geocoding place names to coordinates
- Handling location ambiguity (“Paris” = France or Texas?)
- Creating meaningful geographic visualizations
- Respecting privacy implications of location tracking
Key Concepts: Geocoding, choropleth maps, Folium, spatial clustering
| Difficulty: Intermediate-Advanced | Time Estimate: 12-16 hours |
Prerequisites: Completed Projects 1-5, geographic/mapping basics
Real World Outcome
$ python geo_analyzer.py --input tweets.db --output geo_report/
[2024-01-15 22:00:00] Loading tweets with location data...
[2024-01-15 22:00:05] 49,766 tweets total
- With GPS coordinates: 2,345 (4.7%)
- With user location: 28,456 (57.2%)
- With place mention: 12,345 (24.8%)
Geocoding user locations...
[2024-01-15 22:02:00] Successfully geocoded 24,567 locations
Geographic Distribution:
┌─────────────────┬──────────┬──────────┬───────────────┐
│ Region │ Tweets │ % Total │ Avg Sentiment │
├─────────────────┼──────────┼──────────┼───────────────┤
│ United States │ 15,234 │ 38.4% │ +0.12 │
│ - California │ 4,567 │ 11.5% │ +0.18 │
│ - New York │ 3,234 │ 8.1% │ +0.08 │
│ - Texas │ 2,345 │ 5.9% │ +0.15 │
│ United Kingdom │ 4,567 │ 11.5% │ +0.05 │
│ India │ 3,456 │ 8.7% │ +0.22 │
│ Germany │ 2,345 │ 5.9% │ -0.03 │
└─────────────────┴──────────┴──────────┴───────────────┘
Anomalies:
⚠️ Germany showing negative sentiment (investigate)
⚠️ High activity in São Paulo (unexpected market)
Outputs:
- geo_report/choropleth_volume.html (interactive map)
- geo_report/choropleth_sentiment.html (sentiment by region)
- geo_report/cluster_analysis.png (geographic clusters)
The Core Question You’re Answering
Where are your users/customers/mentions coming from, and how does location correlate with sentiment and engagement?
Concepts You Must Understand First
- What percentage of social media posts have precise GPS coordinates?
-
How do you handle a user location of “NYC SF Global”? - What is geocoding and what are its limitations?
- What is a choropleth map and when is it appropriate?
Project 17: Real-Time Streaming Pipeline
What You’ll Build: A real-time streaming data pipeline that connects to social media streaming APIs, processes posts as they arrive, updates metrics in real-time, and triggers alerts when anomalies are detected.
Why It Teaches the Concept: Batch analysis tells you what happened. Streaming analysis tells you what is happening NOW. For crisis monitoring, brand protection, and trend surfing, real-time matters. This project teaches you stream processing fundamentals.
Core Challenges:
- Connecting to streaming APIs (filtered streams)
- Processing data faster than it arrives (backpressure)
- Maintaining stateful aggregations in-memory
- Detecting anomalies in streaming context
- Persisting results without blocking the stream
Key Concepts: Streaming APIs, event processing, windowed aggregation, backpressure
| Difficulty: Advanced | Time Estimate: 16-20 hours |
Prerequisites: Async Python (asyncio), completed Projects 1-6
Real World Outcome
$ python stream_monitor.py --track "productname,@company,#brand"
[2024-01-15 23:00:00] Connecting to X Filtered Stream API...
[2024-01-15 23:00:01] Stream connected. Tracking: productname, @company, #brand
[2024-01-15 23:00:01] Real-time dashboard: http://localhost:8501
LIVE METRICS (updating every 5 seconds):
┌────────────────────────────────────────────────────────────────┐
│ Current Rate: 12.3 tweets/min (baseline: 8.5) [NORMAL] │
│ Last 5 min: 62 tweets │
│ Sentiment: +0.23 (5-min avg) │
│ Top Hashtags: #productname (45), #tech (23), #review (18) │
└────────────────────────────────────────────────────────────────┘
[23:05:23] Rate spike detected: 45.6 tweets/min (5x normal)
[23:05:24] ALERT: Volume anomaly - possible viral content or crisis
[23:05:24] Sample tweet: "OMG @company just announced..."
[23:05:24] Sentiment still positive (+0.31) - likely good news
[23:10:00] Rate returning to normal: 14.2 tweets/min
[23:10:00] Alert resolved.
The Core Question You’re Answering
How do you process, analyze, and react to social media data in real-time, as events are happening?
Concepts You Must Understand First
- What is the difference between polling an API and using a streaming API?
- What is backpressure and why does it matter?
- How do you maintain running averages/counts without storing all data?
- What is a tumbling window vs sliding window?
The Interview Questions They’ll Ask
- “Your stream is receiving 1,000 tweets/second but you can only process 500/second. What do you do?”
- “How would you detect a trending hashtag in a streaming context?”
- “What happens to your in-memory state if your process crashes?”
- “How would you scale this to handle 10x the volume?”
Project 18: Interactive Dashboard with Streamlit/Dash
What You’ll Build: A full-featured interactive web dashboard for social media analytics that allows users to filter data, explore visualizations, and drill down into details, with auto-refresh for near-real-time updates.
Why It Teaches the Concept: Command-line tools and static reports are insufficient for stakeholders who want to explore data interactively. This project teaches you to build user-facing data applications that make your analysis accessible to non-technical users.
Core Challenges:
- Designing intuitive UI for data exploration
- Handling large datasets without freezing the browser
- Implementing efficient caching for repeated queries
- Creating responsive visualizations that update with filters
- Deploying for stakeholder access
Key Concepts: Streamlit, reactive programming, caching, session state
| Difficulty: Intermediate-Advanced | Time Estimate: 14-18 hours |
Prerequisites: Completed Projects 4-11
Real World Outcome
$ streamlit run dashboard.py
You can now view your Streamlit app in your browser.
Local URL: http://localhost:8501
Network URL: http://192.168.1.5:8501
Dashboard Features:
┌─────────────────────────────────────────────────────────────────────────┐
│ SOCIAL MEDIA ANALYTICS DASHBOARD │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ [Sidebar] [Main Content] │
│ ┌──────────────────┐ ┌────────────────────────────────────┐│
│ │ Date Range │ │ Volume Over Time ││
│ │ [Jan 1 - Jan 15] │ │ [Interactive line chart] ││
│ │ │ │ ││
│ │ Sentiment Filter │ │ Sentiment Distribution ││
│ │ [x] Positive │ │ [Pie chart] ││
│ │ [x] Neutral │ │ ││
│ │ [x] Negative │ │ Top Hashtags │ Top Users ││
│ │ │ │ [Bar chart] │ [Table] ││
│ │ Hashtag Filter │ │ ││
│ │ [Search...] │ │ Geographic Map ││
│ │ │ │ [Choropleth] ││
│ │ [Export CSV] │ │ ││
│ │ [Refresh Data] │ │ Raw Data (first 100 rows) ││
│ └──────────────────┘ │ [Scrollable table] ││
│ └────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────────────────┘
The Core Question You’re Answering
How do you make your analysis explorable and accessible to stakeholders who can’t write code?
Concepts You Must Understand First
- What is reactive programming and how does Streamlit implement it?
- How does
@st.cache_dataprevent redundant computation? - What is session state and when do you need it?
- How do you handle long-running computations without blocking the UI?
The Interview Questions They’ll Ask
- “Your dashboard takes 30 seconds to load. How do you optimize it?”
- “How would you implement user authentication for sensitive data?”
- “The CEO wants to see this on their phone. How do you handle mobile?”
- “How would you add real-time updates without constant page refreshes?”
Hints in Layers
Hint 1: Start with streamlit and layout widgets with st.sidebar, st.columns(). Use st.plotly_chart() for interactive plots.
Hint 2: Cache data loading with @st.cache_data. Cache expensive computations with proper hash_funcs.
Hint 3: Use st.session_state to persist user selections across reruns. Initialize defaults in a check.
Hint 4: For auto-refresh, use st_autorefresh from streamlit-autorefresh package or time.sleep() with st.rerun().
Project 19: Cross-Platform Data Aggregator
What You’ll Build: A unified data aggregation system that collects, normalizes, and combines data from multiple social platforms (X, Reddit, YouTube comments, Mastodon) into a single analysis-ready format.
Why It Teaches the Concept: Real social listening spans multiple platforms. Your audience is not just on one network. This project teaches you to design flexible data models that accommodate different platforms’ data structures while enabling cross-platform analysis.
Core Challenges:
- Designing a unified schema for heterogeneous data
- Mapping platform-specific fields to common fields
- Handling different ID formats and timestamps
- Reconciling different engagement metric definitions
- Managing multiple API credentials and rate limits
Key Concepts: Data normalization, schema design, ETL pipelines, platform abstraction
| Difficulty: Advanced | Time Estimate: 16-20 hours |
Prerequisites: Completed Projects 1-3
Real World Outcome
$ python multi_platform_collector.py --query "climate change" --platforms x,reddit,youtube
[2024-01-16 00:00:00] Starting multi-platform collection...
Platform: X (Twitter)
[00:00:01] Authenticating...
[00:05:00] Collected 2,345 tweets
Platform: Reddit
[00:05:01] Authenticating...
[00:10:00] Collected 1,234 posts + 5,678 comments
Platform: YouTube
[00:10:01] Authenticating...
[00:15:00] Collected comments from 234 videos
Normalization:
[00:15:01] Mapping X data to unified schema...
[00:15:02] Mapping Reddit data to unified schema...
[00:15:03] Mapping YouTube data to unified schema...
Unified Dataset:
┌────────────────────────────────────────────────────────────────┐
│ Total records: 9,491 │
├─────────────┬────────────┬────────────────────────────────────┤
│ Platform │ Count │ Unique Authors │
├─────────────┼────────────┼────────────────────────────────────┤
│ X │ 2,345 │ 1,987 │
│ Reddit │ 6,912 │ 4,234 │
│ YouTube │ 234 │ 189 │
└─────────────┴────────────┴────────────────────────────────────┘
Unified Schema Fields:
- id (platform_recordid)
- platform (x|reddit|youtube)
- author_id, author_name
- content_text
- created_at (UTC)
- engagement_count (likes+shares+comments normalized)
- parent_id (for replies/comments)
- url
Cross-Platform Insights:
- Topic most discussed on: Reddit (58.7%)
- Highest engagement: YouTube (23.4 avg per post)
- Sentiment comparison: X (+0.05), Reddit (-0.12), YouTube (+0.18)
Saved to: unified_climate_change.parquet
The Core Question You’re Answering
How do you create a single source of truth from multiple platforms with different data models, metrics, and semantics?
Concepts You Must Understand First
- What is the difference between a “post” on X vs Reddit vs YouTube?
- How do you compare “likes” across platforms with different norms?
- What is the principle of “schema on read” vs “schema on write”?
- How do you handle platform-specific fields that don’t exist elsewhere?
Project 20: Capstone - Full Social Listening Platform
What You’ll Build: A production-ready social listening platform that combines all previous projects into an integrated system with scheduled collection, automated analysis, alerting, and a stakeholder-facing dashboard.
Why It Teaches the Concept: Real-world systems are not single scripts—they are integrated pipelines with scheduling, error handling, monitoring, and user interfaces. This capstone project teaches you to think like a software engineer building a product, not just a data scientist writing notebooks.
Core Challenges:
- Designing a modular architecture that uses previous projects as components
- Implementing scheduling (hourly collection, daily reports)
- Adding monitoring and alerting (errors, anomalies)
- Creating configuration for different clients/use cases
- Deploying for production use
Key Concepts: System architecture, scheduling, monitoring, deployment
| Difficulty: Advanced | Time Estimate: 30-40 hours |
Prerequisites: All previous projects completed
Real World Outcome
$ python run_platform.py --config config/brand_monitoring.yaml
[2024-01-16 08:00:00] Social Listening Platform v1.0
[2024-01-16 08:00:00] Loading configuration: brand_monitoring.yaml
Configuration:
Client: TechCorp Inc
Keywords: ["techcorp", "@techcorp", "#techcorpproduct"]
Platforms: [X, Reddit]
Collection: Every 15 minutes
Analysis: Hourly sentiment, daily reports
Alerts: Volume spikes, negative sentiment, competitor mentions
Services Started:
✓ Scheduler (APScheduler)
✓ Data Collector (X API, Reddit API)
✓ Analysis Pipeline (Sentiment, Topics, Influencers)
✓ Alert System (Slack, Email)
✓ Dashboard (Streamlit, port 8501)
✓ API Server (FastAPI, port 8000)
[08:00:01] First collection cycle starting...
[08:15:00] Collection complete: 234 new posts
[08:15:01] Analysis complete: Sentiment +0.15, 3 new topics
[08:15:02] No alerts triggered.
[09:00:00] Hourly report generated: reports/hourly_20240116_0900.pdf
[09:00:01] Report sent to: marketing@techcorp.com
Dashboard: http://localhost:8501
API Docs: http://localhost:8000/docs
Logs: logs/platform_20240116.log
System architecture:
┌─────────────────────────────────────────────────────────────────────────┐
│ SOCIAL LISTENING PLATFORM │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌────────────────┐ │
│ │ Scheduler │ │
│ │ (APScheduler) │ │
│ └───────┬────────┘ │
│ │ │
│ ▼ │
│ ┌────────────────┐ ┌────────────────┐ ┌────────────────┐ │
│ │ X Collector │ │Reddit Collector│ │ YouTube Coll. │ │
│ └───────┬────────┘ └───────┬────────┘ └───────┬────────┘ │
│ │ │ │ │
│ └─────────────────────┴─────────────────────┘ │
│ │ │
│ ▼ │
│ ┌────────────────────┐ │
│ │ Unified Database │ │
│ │ (PostgreSQL) │ │
│ └─────────┬──────────┘ │
│ │ │
│ ┌────────────────────┼────────────────────┐ │
│ ▼ ▼ ▼ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Sentiment │ │ Topics │ │ Influencer │ │
│ │ Analyzer │ │ Modeler │ │ Finder │ │
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
│ │ │ │ │
│ └───────────────────┴───────────────────┘ │
│ │ │
│ ▼ │
│ ┌────────────────────┐ │
│ │ Alert Engine │───────▶ Slack/Email │
│ └────────────────────┘ │
│ │ │
│ ┌──────────────────┴──────────────────┐ │
│ ▼ ▼ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Dashboard │ │ API Server │ │
│ │ (Streamlit) │ │ (FastAPI) │ │
│ └──────────────────┘ └──────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
The Core Question You’re Answering
How do you build a production-grade data platform that runs reliably, scales to handle growth, and provides value to stakeholders?
Concepts You Must Understand First
- What is the difference between a script and a service?
- How do you handle failures gracefully (retry, circuit breaker, dead letter queue)?
- What is the role of configuration files vs hardcoded values?
- How do you monitor system health (logs, metrics, alerts)?
The Interview Questions They’ll Ask
- “Walk me through your system architecture. Why did you make these design choices?”
- “The X API is down. How does your system handle this?”
- “A new client wants to track a different set of keywords. How quickly can you onboard them?”
- “Your database is filling up. How do you implement data retention policies?”
- “How would you scale this to handle 10 clients simultaneously?”
Project Comparison Table
| # | Project | Difficulty | Time | Key Skills | Data Engineering | NLP | Network | ML |
|---|---|---|---|---|---|---|---|---|
| 1 | X API Collector | Beginner-Int | 8-12h | API, Auth | ★★★ | |||
| 2 | Reddit Scraper | Beginner-Int | 6-10h | PRAW, Trees | ★★★ | |||
| 3 | Sheets Aggregator | Beginner-Int | 6-8h | gspread, ETL | ★★★ | |||
| 4 | Pandas Cleaning | Intermediate | 8-12h | pandas, regex | ★★★ | ★ | ||
| 5 | Sentiment Analysis | Intermediate | 10-14h | VADER, TextBlob | ★ | ★★★ | ||
| 6 | Trending Detection | Int-Advanced | 12-16h | Statistics | ★★ | ★★ | ||
| 7 | Network Analysis | Int-Advanced | 14-18h | NetworkX | ★ | ★★★ | ||
| 8 | Hashtag Analysis | Intermediate | 8-12h | Co-occurrence | ★★ | ★ | ★★ | |
| 9 | Engagement Metrics | Intermediate | 8-12h | Metrics design | ★★★ | |||
| 10 | Visualization | Intermediate | 10-14h | Matplotlib | ★★★ | |||
| 11 | Report Generator | Intermediate | 8-12h | Jupyter, nbconvert | ★★★ | |||
| 12 | Time Series | Int-Advanced | 12-16h | statsmodels | ★★ | ★★ | ||
| 13 | Bot Detection | Advanced | 16-20h | Classification | ★ | ★ | ★ | ★★★ |
| 14 | Influencer ID | Advanced | 14-18h | Ranking, PageRank | ★ | ★ | ★★★ | ★ |
| 15 | Topic Modeling | Advanced | 14-18h | LDA, Gensim | ★ | ★★★ | ★★ | |
| 16 | Geographic | Int-Advanced | 12-16h | Geocoding, Folium | ★★★ | |||
| 17 | Streaming | Advanced | 16-20h | Async, Real-time | ★★★ | |||
| 18 | Dashboard | Int-Advanced | 14-18h | Streamlit | ★★★ | |||
| 19 | Multi-Platform | Advanced | 16-20h | ETL, Schema | ★★★ | |||
| 20 | Capstone | Advanced | 30-40h | Architecture | ★★★ | ★★ | ★★ | ★★ |
Recommended Learning Paths
Path A: Marketing Analyst (10-12 weeks)
For those who want to support marketing teams with social insights:
Week 1-2: Projects 1-3 (Data Collection) Week 3-4: Projects 4-5 (Cleaning, Sentiment) Week 5-6: Projects 8-9 (Hashtags, Engagement) Week 7-8: Projects 10-11 (Visualization, Reports) Week 9-10: Project 18 (Dashboard) Week 11-12: Customize for your organization
Path B: Data Scientist (14-16 weeks)
For those building advanced analytics capabilities:
Week 1-2: Projects 1-2 (APIs) Week 3-4: Projects 4-5 (Cleaning, Sentiment) Week 5-6: Projects 6, 12 (Trends, Time Series) Week 7-8: Projects 7, 14 (Networks, Influencers) Week 9-10: Projects 13, 15 (Bots, Topics) Week 11-12: Project 17 (Streaming) Week 13-14: Project 19-20 (Multi-platform, Capstone)
Path C: Platform Engineer (16-20 weeks)
For those building production social listening systems:
Week 1-3: Projects 1-4 (Data Pipeline Foundation) Week 4-5: Projects 5-6 (Analysis Basics) Week 6-8: Projects 10-12 (Visualization, Reports, Time Series) Week 9-10: Project 17 (Streaming) Week 11-12: Project 18 (Dashboard) Week 13-14: Project 19 (Multi-Platform) Week 15-18: Project 20 (Capstone Platform) Week 19-20: Production hardening and deployment
Summary
| # | Project | Skills Learned | Career Relevance |
|---|---|---|---|
| 1 | X API Collector | OAuth, rate limiting, pagination | Foundation for all API work |
| 2 | Reddit Scraper | PRAW, tree traversal | Multi-platform capability |
| 3 | Sheets Aggregator | Google API, stakeholder communication | Business reporting |
| 4 | Pandas Cleaning | Data quality, normalization | 80% of data science work |
| 5 | Sentiment Analysis | NLP, VADER, ensemble methods | Most requested analysis |
| 6 | Trending Detection | Anomaly detection, statistics | Real-time monitoring |
| 7 | Network Analysis | Graph theory, centrality | Understanding social structure |
| 8 | Hashtag Analysis | Co-occurrence, clustering | Topic discovery |
| 9 | Engagement Metrics | KPI design, benchmarking | Marketing analytics |
| 10 | Visualization | Chart selection, storytelling | Communication |
| 11 | Report Generator | Reproducibility, automation | Operational efficiency |
| 12 | Time Series | Decomposition, forecasting | Trend analysis |
| 13 | Bot Detection | Classification, feature engineering | Data quality |
| 14 | Influencer ID | Ranking algorithms, multi-criteria | Influencer marketing |
| 15 | Topic Modeling | LDA, unsupervised learning | Content analysis |
| 16 | Geographic | Geocoding, spatial analysis | Market analysis |
| 17 | Streaming | Real-time processing | Crisis monitoring |
| 18 | Dashboard | Interactive applications | Stakeholder enablement |
| 19 | Multi-Platform | Schema design, ETL | Enterprise integration |
| 20 | Capstone | System architecture | Production readiness |