Project 11: ScreenLens Capture and OCR Lab

Build a visible, user-consented screen-inspection session that performs bounded local OCR, relates pixels to accessibility semantics, and proves that every stop path releases capture resources.

Quick Reference

Attribute	Value
Difficulty	Level 3 - Advanced
Time Estimate	22-32 hours
Language	Kotlin (alternatives: Java; C++ for an optional image-processing experiment)
Prerequisites	Projects 1, 3, 8, and 9; Android service lifecycle; coroutines; accessibility-node inspection; image-buffer basics
Key Topics	MediaProjection, foreground-service types, VirtualDisplay, ImageReader, frame backpressure, local OCR, privacy, semantic reconciliation

1. Learning Objectives

By completing this project, you will:

Explain why MediaProjection grants session-scoped authority rather than a reusable screen-capture permission.
Model requesting, active, stopping, stopped, and failed capture states explicitly.
Own MediaProjection, foreground-service, VirtualDisplay, ImageReader, frame, and OCR lifetimes without leaking any of them.
Bound frame rate, dimensions, memory, queue depth, and OCR concurrency.
Recreate size-dependent resources after rotation or selected-app window resizing.
Compare OCR candidates with accessibility nodes without allowing pixels to silently override stronger semantic evidence.
Minimize captured data by processing frames in memory and persisting only redacted diagnostic metadata by default.
Handle system stop, app stop, lock, callback, process death, and pipeline failure through one idempotent cleanup path.
Test FLAG_SECURE, unavailable semantics, ambiguous OCR, and revoked accessibility state as explicit non-actionable outcomes.
Produce a policy and privacy explanation suitable for a real screen-inspection feature.

2. Theoretical Foundation

2.1 Core Concepts

Capture authority is a live session. MediaProjection begins only after Android presents a system-controlled consent flow and the user selects what may be shared. The result is not a durable permission to capture later. Modern Android requires consent for each projection session, and a projection token is intended for one active capture lifecycle. The product must therefore treat Idle -> Requesting -> Active as a user-visible transition, never restore an old token after process death, and never start capture from a background surprise. The authority ends when the user stops sharing, the system revokes it, the app stops it, or the process disappears.

Foreground execution is part of the contract. A capture session is immediate, user-perceptible work. On modern target SDKs, the app declares the mediaProjection foreground-service type and its related permissions, starts the service in the allowed order, and exposes an ongoing notification with a stop action. The foreground service does not make the process immortal. It makes the work visible and gives Android a better lifecycle contract. State still has to survive or terminate safely when the system kills components.

VirtualDisplay is a resource graph. MediaProjection supplies captured display content to a VirtualDisplay, which renders into a Surface commonly owned by an ImageReader. Each frame may reference substantial native memory. A 1440 x 3200 RGBA frame is roughly 17.6 MiB before OCR intermediates. Holding several frames, bitmaps, crops, and model tensors can exhaust the heap quickly. Ownership must be explicit: acquire the newest image, copy or crop only what is needed, close the image promptly, and release the Surface, reader, display, projection callback, and service state in reverse order.

Backpressure is a correctness property. Screen content can arrive much faster than OCR can process it. Queueing every frame creates stale analysis, memory growth, heat, and battery drain. ScreenLens is an inspector, not a video archive, so “latest frame wins” is the appropriate default. At most one OCR job runs and one newer frame may replace the pending candidate. Dropped-frame counters are healthy evidence that pressure is controlled, not necessarily an error.

Pixels and semantics are different evidence sources. OCR observes visible glyph-like pixels and returns text, boxes, confidence, and sometimes language hints. Accessibility exposes a semantic tree containing class, text, content description, bounds, actions, and state when an app provides it. OCR can see custom-rendered text that is missing from the semantic tree; accessibility can distinguish actionable controls whose visual appearance is ambiguous. A safe inspector presents both, calculates overlaps, and labels disagreement. It does not automatically tap an OCR box merely because a string was recognized.

Privacy starts before storage. A frame may contain messages, account balances, health data, tokens, photographs, or one-time codes. “We delete screenshots later” is weaker than never serializing them. The default pipeline keeps frames in memory, crops only user-selected regions, performs OCR locally, redacts recognized text in diagnostics, and persists geometry plus confidence only when needed. Debug screenshot export is a separate, explicit, time-limited user action with a preview and deletion control.

Secure and ambiguous content are terminal states. Android may suppress protected content from capture. OCR may also return low-confidence or conflicting candidates. These are not invitations to bypass the platform or guess. The correct results are ProtectedContent, NoVisualEvidence, or AmbiguousEvidence, with no automated action. ScreenLens complements semantic automation; it does not defeat secure surfaces, permission dialogs, CAPTCHAs, credentials, purchases, or consent flows.

2.2 Why This Matters

Many automation products begin with coordinates because pixels appear universal. Coordinates fail under density, rotation, font scale, localization, windowing, animations, and app redesign. OCR improves visual interpretation but introduces lifecycle, privacy, performance, and confidence problems. This lab teaches how to use visual evidence honestly: as a user-visible, bounded capability that supplements semantics.

The same engineering applies to screen sharing, remote support, document capture, visual testing, assistive inspection, and on-device computer vision. The learner practices resource ownership, backpressure, privacy classification, asynchronous stop handling, and evidence fusion—skills that matter well beyond automation.

2.3 Historical Context / Background

Android screen capture evolved away from privileged framebuffer access toward MediaProjection, a user-mediated API. Newer Android releases tightened the relationship among user consent, foreground-service declarations, and capture lifetime. Android 14 guidance requires fresh consent for each session and the appropriate media-projection foreground-service setup. Android 15 prevents launching a media-projection foreground service from BOOT_COMPLETED, reinforcing that capture must be contemporaneous and user-visible rather than a boot-persistent observer.

OCR has likewise shifted from server-only recognition toward efficient on-device models. Local processing reduces network exposure and latency, but it does not remove privacy responsibility: sensitive pixels still pass through app memory, diagnostics can leak recognized text, and bundled or downloaded models have version and resource costs.

2.4 Common Misconceptions

“The consent result can be cached forever.” It represents one session and must not be treated as durable background authority.
“A foreground service makes capture survive anything.” The process can still die, and capture must stop safely rather than resurrect silently.
“More frames improve OCR.” Excess frames often create stale work, memory pressure, heat, and worse responsiveness.
“OCR text identifies a clickable control.” Recognition supplies visual evidence, not action semantics or authorization.
“Local OCR means there is no privacy risk.” Frames and recognized strings remain sensitive inside memory, logs, traces, and exports.
“A black secure frame should be worked around.” Protected content is a boundary to respect.
“Cleanup can be spread across callbacks.” Multiple stop sources require one centralized idempotent release routine.

3. Project Specification

3.1 What You Will Build

Build ScreenLens as a lab-only feature connected to the owned Sandbox Target application from earlier projects. Its home screen has a prominent Start ScreenLens button, a privacy summary, a frame-rate selector capped at low inspection rates, and a region mode defaulting to “user-selected app/window.”

After the system consent flow, the active screen contains:

A clearly labeled low-rate preview.
Current session state and elapsed time.
Frame dimensions, acquisition rate, dropped count, and OCR duration.
OCR boxes with confidence categories rather than misleading precision.
Accessibility-node outlines when the inspector service is enabled.
A reconciliation panel showing semantic match, visual-only candidate, semantic-only node, or disagreement.
A persistent Stop control mirrored in the foreground notification.
A no-history-by-default statement and a diagnostic metadata toggle.

The application must never capture before consent, automatically restart a stopped session, archive raw frames by default, or convert OCR evidence into an unattended tap.

3.2 Functional Requirements

Fresh consent: Launch the system MediaProjection chooser for every new session.
Explicit session state: Persist only non-authority UI facts; never persist or reuse the projection token.
Typed foreground service: Declare and start the media-projection service according to the current target SDK contract.
Visible termination: Provide app, notification, and system stop paths.
Bounded acquisition: Keep the latest frame only and expose drop/backpressure metrics.
Rotation handling: Recalculate density and dimensions, then resize or recreate dependent resources safely.
Local OCR: Process only in memory with an on-device model and no network fallback.
Semantic correlation: Compare OCR geometry to the latest accessibility snapshot by package, window, and overlap.
Confidence policy: Classify results as supported, conflicting, low-confidence, protected, or absent.
Privacy controls: Store no raw screenshot history by default; redact recognized text in logs.
Idempotent cleanup: Route every stop and failure path through the same release operation.
Forbidden-target policy: Refuse automated use on credentials, security dialogs, permission grants, purchases, and protected content.

3.3 Non-Functional Requirements

Performance: Default inspection rate at or below two analyzed frames per second; no unbounded queues.
Memory: Stable native and managed memory during a ten-minute session and repeated rotation.
Reliability: Stop completes even if OCR, preview, or accessibility integration is unhealthy.
Privacy: No raw frame or recognized secret appears in durable logs or support bundles.
Usability: Capture state and stop controls remain obvious at all times.
Compatibility: Record API level, target SDK, form factor, selection mode, and OCR model version.
Policy: Document that the feature does not evade secure content or silently capture in the background.

3.4 Example Usage / Output

ScreenLens / Active Session
Scope: Sandbox Target app window
State: ACTIVE             Elapsed: 00:01:42
Frame: 1080 x 2400        Analysis: 1.8 fps
Dropped by latest-wins: 96
OCR p95: 142 ms           Raw frames saved: 0

Candidate A
Text: [redacted, 12 chars] Confidence: HIGH
Bounds: (112, 418)-(706, 520)
Semantic overlap: Button node, 0.91 IoU
Decision: SUPPORTED_VISUAL_AND_SEMANTIC

Candidate B
Text: [redacted, 8 chars] Confidence: LOW
Semantic overlap: none
Decision: REVIEW_ONLY

[Stop ScreenLens]

3.5 Real World Outcome

Install ScreenLens and the Sandbox Target on a test device. Open ScreenLens and read the concise privacy disclosure before tapping Start ScreenLens. Android—not the app—shows the capture chooser. Select only the sandbox app or test display. Confirm that the system capture indicator and ScreenLens foreground notification appear before any preview.

Open a sandbox page containing one standard button, one custom-drawn label, one intentionally low-contrast word, and a simulated protected panel. ScreenLens overlays OCR rectangles and accessibility-node bounds. The standard button appears as a supported semantic/visual match. The custom label appears visual-only and review-only. The low-confidence word is visibly marked ambiguous. The protected panel yields no actionable content.

Rotate the device repeatedly and resize the selected app where supported. Frame timestamps continue, dimensions update, and memory returns to its steady range. Stop once from the notification, once from the system capture UI, once from the app, and once by forcing process death. After each stop, the preview freezes immediately, the notification disappears, no further frames arrive, and reopening ScreenLens shows Idle rather than resuming capture. The history screen contains only redacted geometry, timing, and confidence statistics.

4. Solution Architecture

4.1 High-Level Design

 User tap
    |
    v
+-------------------+       system-owned       +---------------------+
| Consent Launcher  | -----------------------> | Capture Chooser     |
+---------+---------+                           +----------+----------+
          | approved result                                |
          v                                                v
+-------------------+    owns session     +--------------------------+
| Session Controller| ------------------> | Projection FGS           |
| state machine     |                     | visible notification     |
+---------+---------+                     +------------+-------------+
          |                                                |
          |                           +--------------------+----------------+
          |                           |                                     |
          v                           v                                     v
+-------------------+       +-------------------+                 +----------------+
| VirtualDisplay    | ----> | ImageReader       | -- latest -->   | Local OCR Port |
| size/density      |       | bounded buffers   |                 | one job max    |
+-------------------+       +-------------------+                 +-------+--------+
                                                                         |
 Accessibility snapshot -------------------------------------------------+
                                                                         v
                                                           +-----------------------+
                                                           | Evidence Reconciler   |
                                                           | overlap + confidence  |
                                                           +-----------+-----------+
                                                                       |
                                                                       v
                                                           preview + redacted facts

 stop callback / notification / app / error / process end
                         |
                         v
              idempotent reverse-order release

4.2 Key Components

Component	Responsibility	Key Decisions
Consent coordinator	Launch and interpret system capture request	Never caches authority or auto-retries denial
Session state machine	Serialize start, active, stop, and error transitions	One session ID; invalid transitions rejected
Projection foreground service	Own visible lifetime and stop notification	No boot start; no hidden resurrection
Capture resource owner	Create and release projection/display/reader/surface	Centralized reverse-order cleanup
Latest-frame gate	Close stale frames and retain only newest work	Bounded queue depth of one
OCR adapter	Convert a selected crop to local recognition candidates	No network fallback; cancellation-aware
Semantic snapshot adapter	Supply current accessibility nodes	Package/window scoping and freshness timestamp
Evidence reconciler	Relate rectangles, strings, roles, and confidence	Never produces an automatic click command
Privacy projector	Convert raw evidence to safe diagnostic facts	Redact before persistence
Session UI	Render preview, metrics, warnings, and stop	Reflects controller state only

4.3 Data Structures

CaptureSession
  session_id: opaque identifier
  scope: display | selected_app
  phase: idle | requesting | active | stopping | stopped | failed
  dimensions: width, height, density
  started_at: monotonic + wall-clock display value
  stop_reason: optional typed reason

FrameLease
  session_id
  frame_sequence
  acquired_at_monotonic
  dimensions
  rotation
  close_state

VisualCandidate
  region
  text_classification: redacted hash/length, never raw in durable record
  confidence_band
  model_version

ReconciledEvidence
  visual_candidate_id
  semantic_node_ref: optional ephemeral reference
  overlap_score
  freshness_delta
  decision: supported | visual_only | semantic_only | conflict | ambiguous | protected

4.4 Algorithm Overview

Key Algorithm: Latest-Frame Evidence Reconciliation

Reject frames whose session ID is no longer active.
Replace and close any pending unprocessed frame.
Acquire one processing lease if no OCR job is running.
Normalize rotation and crop only the inspection region.
Run local OCR with a deadline and cancellation token.
Obtain a same-package, same-window semantic snapshot with a freshness bound.
Calculate geometric overlap and compatible text/role evidence.
Classify support, conflict, ambiguity, or missing evidence.
Render ephemeral overlays and persist only privacy-projected metrics.
Close every image and intermediate buffer in a finally-equivalent path.

Complexity Analysis:

OCR cost depends on model and pixels; pixel preparation is O(width x height) for the selected region.
Naive visual-to-semantic matching is O(V x S), where V is OCR candidates and S is semantic nodes.
A spatial index can reduce candidate comparisons when the tree is large.
Queue space is O(1) frames by design; candidate metadata is O(V + S).

5. Implementation Guide

5.1 Development Environment Setup

Use Android Studio with the repository’s selected compile/target SDK, one emulator, and one physical device if available. Install only an on-device OCR dependency. Configure a sandbox build that never targets third-party credential or payment screens.

Environment verification checklist
- Android Studio and SDK tools report expected versions
- device API and target SDK are recorded
- sandbox target is installed
- accessibility inspector can be enabled independently
- capture notification permission behavior is understood for the API level
- profiler and logcat are available

Review the current official MediaProjection and foreground-service documentation before defining the manifest. Treat documentation snippets as configuration shape, not as a complete implementation.

5.2 Project Structure

screenlens/
|-- app/
|   |-- session/          # consent and state orchestration
|   |-- projection/       # service and resource ownership
|   |-- frames/           # latest-frame gate and transforms
|   |-- ocr/              # local OCR port and adapter
|   |-- semantics/        # accessibility snapshot port
|   |-- reconcile/        # pure evidence matching
|   |-- privacy/          # redaction and retention
|   `-- ui/               # idle, requesting, active, result screens
|-- domain/
|   `-- evidence/         # framework-free models and policies
|-- test-fixtures/
|   `-- sandbox-screens/  # synthetic, non-sensitive fixtures
`-- tests/
    |-- unit/
    |-- integration/
    `-- device/

5.3 The Core Question You’re Answering

“What can pixels reveal when semantics are missing, and how do I use that evidence without pretending screen capture is silent, permanent, or universally permitted?”

The project is successful when your answer includes authority, lifetime, pressure, confidence, privacy, and refusal—not merely how to obtain a bitmap.

5.4 Concepts You Must Understand First

Session-scoped authority
- What exact user action creates a session?
- Which object is authoritative, and when does it become invalid?
- Reference: Android MediaProjection documentation, lifecycle and consent sections.
Foreground-service types
- Which manifest declarations and start order apply to the current target SDK?
- Why is capture not a boot-time service?
- Reference: Android foreground-service type requirements.
Native buffer ownership
- Why can one unclosed Image exhaust memory over time?
- Which layer closes frames and intermediates?
- Book Reference: “Operating Systems: Three Easy Pieces” — persistence/resource-lifetime chapters.
Backpressure
- Why is latest-wins appropriate for inspection but not for archival recording?
- What metrics prove bounded behavior?
- Book Reference: “Release It!, 2nd Edition” — stability and overload patterns.
OCR uncertainty
- What does model confidence omit?
- How do geometry, semantic role, and snapshot freshness change a decision?
Data minimization
- Can the feature work without durable pixels or raw recognized text?
- Book Reference: “Foundations of Information Security” — privacy and risk.

5.5 Questions to Guide Your Design

How will the UI distinguish Requesting from Active before the first frame?
Which component alone owns stop and release?
What happens when stop arrives while OCR is running?
How do you reject a late frame from an earlier session?
What is the maximum region size and analysis rate?
How will rotation be serialized with frame acquisition?
How fresh must an accessibility snapshot be to support a visual candidate?
Which fields are permitted in logs, metrics, history, and exports?
What evidence proves that no frame reached disk?
How does the app explain protected or ambiguous content without suggesting bypasses?

5.6 Thinking Exercise

Draw the Ownership and Stop Graph

On paper, draw consent result, session controller, service, MediaProjection, callback, VirtualDisplay, Surface, ImageReader, acquired Image, crop buffer, OCR job, preview, and history projector.

Trace these events independently:

User taps the in-app stop button.
User taps the notification stop action.
Android stops capture from system UI.
Rotation arrives during OCR.
OCR throws while a frame is leased.
Accessibility disconnects.
The process dies after consent but before the first frame.

For each trace, identify the state transition, cancellation signal, release order, visible user outcome, and whether anything can restart without new consent.

5.7 The Interview Questions They’ll Ask

“Why does MediaProjection require user consent for each session on modern Android?”
“How do foreground-service types affect a screen-capture app?”
“How would you prevent ImageReader backpressure and native-memory leaks?”
“What changes when the selected app rotates or enters multi-window?”
“How do OCR and accessibility semantics complement one another?”
“Why is OCR confidence insufficient for unattended UI automation?”
“How would you prove that a capture product does not persist sensitive frames?”
“What should happen when FLAG_SECURE content is encountered?”

5.8 Hints in Layers

Hint 1: Make invalid transitions impossible

Write the session transition table before creating Android resources. A second start while active should be rejected or routed through a deliberate stop-then-request flow.

Hint 2: Close on replacement

The latest-frame slot owns at most one pending frame. Replacing it closes the older lease immediately.

Hint 3: Tag all asynchronous work

Every callback, frame, and OCR result carries a session ID. Late work from a stopped session is discarded and closed.

Hint 4: Reconcile as a pure function

Feed recorded rectangles and semantic snapshots into a framework-free matcher. This keeps safety rules testable without capturing a screen.

Hint 5: Instrument ownership

Count created and released readers, displays, images, and OCR jobs in debug builds. The active count should return to zero after every stop.

5.9 Books That Will Help

Topic	Book	Chapter/Section
Resource lifetime	“Operating Systems: Three Easy Pieces”	Processes, concurrency, and persistence chapters
Stability under load	“Release It!, 2nd Edition”	Stability patterns and resource management
Visual evidence	“Computer Vision: Algorithms and Applications”	Image formation, features, and recognition
Data minimization	“Foundations of Information Security”	Privacy, risk, and access control
Boundary design	“Clean Architecture”	Component boundaries and policy

5.10 Implementation Phases

Goals:

Complete the session state machine and visible start/stop experience.
Own a projection session without OCR.

Tasks:

Define transitions, stop reasons, and invalid-state behavior.
Add current target-SDK manifest and foreground-service configuration.
Launch the system chooser from a user action.
Surface active state in app and notification.
Implement one idempotent cleanup coordinator.

Checkpoint: Start and stop twenty sessions through all available stop surfaces; every resource count returns to zero and no session restarts itself.

Phase 2: Bounded Frame and OCR Pipeline (8-12 hours)

Goals:

Acquire low-rate frames with constant queue space.
Produce ephemeral local OCR candidates.

Tasks:

Calculate dimensions and density from selected scope.
Create the display/reader surface graph.
Implement latest-wins leasing and cancellation.
Add local OCR behind a port with timeout and metrics.
Recreate size-dependent resources after rotation.

Checkpoint: A ten-minute rotation stress run has bounded memory, current timestamps, expected drop counts, and zero stored screenshots.

Phase 3: Semantic Reconciliation and Privacy Proof (8-12 hours)

Goals:

Compare visual and semantic evidence safely.
Demonstrate redaction, refusal, and supportability.

Tasks:

Snapshot selected accessibility nodes with freshness metadata.
Implement pure overlap/conflict classification.
Render review-only overlays and disagreements.
Add protected/forbidden-target refusal states.
Build redacted metrics history and export preview.
Complete lifecycle, privacy, and policy tests.

Checkpoint: The golden sandbox demo distinguishes supported, visual-only, semantic-only, ambiguous, and protected cases without issuing any automatic action.

5.11 Key Implementation Decisions

Decision	Options	Recommendation	Rationale
Frame policy	Queue all, sample timer, latest-wins	Latest-wins with a low analysis cap	Keeps evidence current and memory bounded
OCR location	Server, hybrid, on-device	On-device only for this lab	Avoids transmitting captured pixels
History	Raw screenshots, raw text, redacted metadata	Redacted metadata by default	Supports diagnosis without surveillance
Semantics relationship	OCR replaces tree, tree replaces OCR, evidence fusion	Evidence fusion with semantic preference	Preserves roles/actions and exposes disagreement
Stop ownership	Each component releases itself, centralized coordinator	Centralized idempotent coordinator	Makes concurrent stop sources safe
Capture restart	Restore after death, boot restart, fresh user start	Fresh user start only	Matches session-scoped consent and visibility

6. Testing Strategy

6.1 Test Categories

Category	Purpose	Examples
Pure unit	Verify transitions and reconciliation	Invalid start, stale session, overlap/conflict bands
Resource integration	Verify ownership and cancellation	Frame replacement closes old image; OCR cancellation closes crop
Device lifecycle	Exercise Android stop sources	System stop, notification stop, lock, rotation, process death
Performance	Prove bounded pressure	Ten-minute session, high-refresh source, repeated resizing
Privacy	Detect forbidden persistence	Filesystem scan, log scan, support-bundle seeded secrets
Compatibility	Record platform differences	Multiple API levels, selected-app/display modes, OEM device

6.2 Critical Test Cases

Consent denied: Remain idle; create no service, display, reader, or frame.
Fresh session: A second capture requires a new system chooser.
Duplicate start: Reject while active without creating another resource graph.
System stop: Callback triggers cleanup and zero later frames.
Concurrent stop: App and notification stop together; cleanup runs effectively once.
Rotation storm: Dimensions update while image ownership remains balanced.
Slow OCR: New frames replace stale work; queue depth never grows.
Late result: OCR from a stopped session cannot update UI or history.
Accessibility loss: Visual candidates remain review-only and clearly labeled.
Protected content: No actionable candidate or bypass suggestion appears.
Seeded secret: Raw fixture secret never appears in logs or export.
Process death: Reopen to Idle and require new consent.

6.3 Test Data

Fixture screen: sandbox-v3
- standard Button: text="RUN SAMPLE", bounds known
- custom canvas label: visual text, no semantic node
- semantic icon button: content description, no visible text
- low-contrast decoy: OCR confidence expected LOW
- moving label: tests stale snapshot rejection
- simulated protected panel: no usable pixels
- seeded secret string: must be redacted before persistence

Expected reconciliation counts:
SUPPORTED=1
VISUAL_ONLY=1
SEMANTIC_ONLY=1
AMBIGUOUS>=1
PROTECTED=1
AUTOMATIC_ACTIONS=0

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

Pitfall	Symptom	Solution
Reusing consent data	Second session throws or violates lifecycle assumptions	Request fresh consent and create a new session graph
Wrong FGS declaration/order	Security or foreground-service exception	Recheck target-SDK-specific MediaProjection documentation
Unclosed Image	Native memory rises until freeze/crash	Lease and close every acquired or replaced frame
Queueing OCR	Preview lags seconds behind and device heats	Latest-wins plus one active OCR job
Stale dimensions	Freeze or stretched frames after rotation	Serialize resize/recreation with session state
Distributed cleanup	Frames continue after one stop surface	Route all stops through one idempotent release path
Raw debug logging	Recognized secrets appear in logcat	Redact before logging or persistence
OCR-as-selector	Wrong control appears “confident”	Require semantic support or explicit user review

7.2 Debugging Strategies

State-transition journal: Record session ID, previous state, event, next state, and redacted stop reason.
Ownership counters: Track active displays, readers, frame leases, and OCR jobs in debug builds.
Timestamp waterfall: Compare acquisition, OCR start/end, semantic snapshot, and render times.
Buffer pressure view: Display received, analyzed, replaced, closed, and late-result counts.
Profiler verification: Observe managed and native memory across rotation and repeated sessions.
No-frame isolation: Test the state/service lifecycle with a fake frame source before debugging OCR.

7.3 Performance Traps

Full-resolution OCR on every display frame is unnecessary for inspection. Crop first, cap dimensions, and analyze at human-inspection rates. Avoid converting the same frame through multiple bitmap formats. Cache model instances only for the active feature lifetime and release large intermediates. Prefer monotonic time for latency. A smooth preview is not evidence of a healthy analyzer; measure acquisition-to-decision age and dropped work directly.

8. Extensions & Challenges

8.1 Beginner Extensions

Add a region-of-interest editor with a visible crop preview.
Add a session summary containing only duration, frame counts, and latency bands.
Add a language/model availability indicator before capture begins.

8.2 Intermediate Extensions

Compare two local OCR engines behind the same port using synthetic fixtures.
Add a spatial index for visual-to-semantic overlap matching.
Add a manual “promote to candidate selector” workflow that still requires later semantic validation.

8.3 Advanced Extensions

Explore app-window capture behavior across foldables, desktop windowing, and multi-display devices.
Build a privacy-preserving visual regression mode using perceptual hashes of owned sandbox screens.
Add hardware-accelerated preprocessing and prove that it improves energy per analyzed frame.
Design an accessibility-tool variant only after writing the distinct purpose, policy, and user-benefit case.

9. Real-World Connections

9.1 Industry Applications

Remote support: Visible session capture with immediate stop and minimization.
Assistive technology: Visual interpretation when an owned workflow lacks useful semantics.
Visual testing: Pixel/semantic comparison against controlled test applications.
Document capture: Bounded image pipelines and local recognition.
Screen sharing: Foreground lifetime, resize handling, and secure-content behavior.

scrcpy: https://github.com/Genymobile/scrcpy - Useful contrast for developer-authorized device mirroring; not an in-app MediaProjection permission model.
ML Kit samples: https://github.com/googlesamples/mlkit - Reference examples for on-device vision API usage; review privacy and lifecycle independently.
AOSP: https://android.googlesource.com/ - Platform source for understanding display and media behavior when public contracts are insufficient.

9.3 Interview Relevance

Android lifecycle questions about foreground services, callbacks, rotation, and process death.
Systems questions about resource ownership, backpressure, and cancellation.
Computer-vision questions about confidence, false positives, and evidence fusion.
Privacy questions about minimization, retention, redaction, and user control.
Product questions about why pixels cannot be a universal hidden automation fallback.

10. Resources

10.1 Essential Reading

Media projection: https://developer.android.com/media/grow/media-projection - Current consent, token, VirtualDisplay, and foreground-service guidance.
Foreground-service types: https://developer.android.com/develop/background-work/services/fgs/service-types - Current type and permission contracts.
Android 14 foreground-service changes: https://developer.android.com/about/versions/14/changes/fgs-types-required - Required type declarations for modern targets.
Android 15 foreground-service changes: https://developer.android.com/about/versions/15/changes/foreground-service-types - Includes boot-start restrictions for media projection.
Secure sensitive activities: https://developer.android.com/privacy-and-security/risks/tapjacking - Context for protected and security-sensitive UI boundaries.

10.2 Video Resources

Android Developers media and privacy sessions: https://www.youtube.com/@AndroidDevelopers - Search the official channel for current MediaProjection and foreground-service talks.
ML Kit official learning resources: https://developers.google.com/ml-kit - Use only the on-device text-recognition material relevant to the selected SDK.

10.3 Tools & Documentation

MediaProjectionManager API: https://developer.android.com/reference/android/media/projection/MediaProjectionManager
MediaProjection API: https://developer.android.com/reference/android/media/projection/MediaProjection
VirtualDisplay API: https://developer.android.com/reference/android/hardware/display/VirtualDisplay
ImageReader API: https://developer.android.com/reference/android/media/ImageReader
ML Kit text recognition: https://developers.google.com/ml-kit/vision/text-recognition/v2/android
Memory Profiler: https://developer.android.com/studio/profile/memory-profiler
Google Play User Data policy: https://support.google.com/googleplay/android-developer/answer/10144311
AccessibilityService policy: https://support.google.com/googleplay/android-developer/answer/10964491

Project 8: Accessibility Inspector supplies semantic snapshots and service-health thinking.
Project 9: Consentful UI Macro Runner defines forbidden actions, postconditions, and emergency stop.
Project 14: Automation Observatory adds fault injection and privacy-safe support evidence.
Project 15: FlowForge Automation Studio integrates ScreenLens as an optional inspection capability.

11. Self-Assessment Checklist

Before considering this project complete, verify:

11.1 Understanding

I can explain why MediaProjection consent and authority are session-scoped.
I can draw every owned resource and its reverse cleanup order.
I can explain latest-wins backpressure and when it would be inappropriate.
I can distinguish OCR evidence from accessibility action semantics.
I can explain why protected or ambiguous content is a refusal state.
I can describe current foreground-service requirements without relying on old assumptions.

11.2 Implementation

All functional and non-functional requirements are demonstrated.
Every new session begins with current system consent.
All stop paths end capture and release every resource.
Rotation and resize tests preserve live frames and bounded memory.
Raw frames and recognized secrets do not persist by default.
OCR never causes an unattended action.
Device, privacy, and compatibility tests pass.

11.3 Growth

I documented one incorrect lifecycle assumption I held before the project.
I can defend the frame-rate, queue, and retention limits quantitatively.
I can explain the architecture in an Android systems interview.
I can identify what would need independent privacy and policy review before release.

12. Submission / Completion Criteria

Minimum Viable Completion:

A fresh-consent capture session against the owned sandbox application.
Visible typed foreground-service operation with app and notification stop controls.
Bounded latest-frame local OCR with no raw screenshot history.
Centralized cleanup proven for normal stop, system stop, and rotation.
A written prohibition list covering credentials, permissions, purchases, and secure content.

Full Completion:

All minimum criteria plus semantic reconciliation, confidence/refusal states, process-death tests, redacted history, and a ten-minute memory/backpressure report.
A compatibility record for at least three API levels or two API levels plus one physical OEM device.
A privacy data-flow diagram and support-bundle secret scan.

Excellence (Going Above & Beyond):

A repeatable performance study comparing crop sizes and OCR models by latency, memory, and energy.
A foldable/multi-window compatibility report with documented resource-recreation behavior.
A reviewed architecture decision record explaining why visual evidence remains inspection-only.

This guide was generated from ANDROID_AUTOMATION_APPS_MASTERY.md. For the complete learning path, see the expanded project index.

Project 11: ScreenLens Capture and OCR Lab

Quick Reference

1. Learning Objectives

2. Theoretical Foundation

2.1 Core Concepts

2.2 Why This Matters

2.3 Historical Context / Background

2.4 Common Misconceptions

3. Project Specification

3.1 What You Will Build

3.2 Functional Requirements

3.3 Non-Functional Requirements

3.4 Example Usage / Output

3.5 Real World Outcome

4. Solution Architecture

4.1 High-Level Design

4.2 Key Components

4.3 Data Structures

4.4 Algorithm Overview

5. Implementation Guide

5.1 Development Environment Setup

5.2 Project Structure

5.3 The Core Question You’re Answering

5.4 Concepts You Must Understand First

5.5 Questions to Guide Your Design

5.6 Thinking Exercise

Draw the Ownership and Stop Graph

5.7 The Interview Questions They’ll Ask

5.8 Hints in Layers

5.9 Books That Will Help

5.10 Implementation Phases

Phase 1: Consent and Lifetime Skeleton (6-8 hours)

Phase 2: Bounded Frame and OCR Pipeline (8-12 hours)

Phase 3: Semantic Reconciliation and Privacy Proof (8-12 hours)

5.11 Key Implementation Decisions

6. Testing Strategy

6.1 Test Categories

6.2 Critical Test Cases

6.3 Test Data

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

7.2 Debugging Strategies

7.3 Performance Traps

8. Extensions & Challenges

8.1 Beginner Extensions

8.2 Intermediate Extensions

8.3 Advanced Extensions

9. Real-World Connections

9.1 Industry Applications

9.2 Related Open Source Projects

9.3 Interview Relevance

10. Resources

10.1 Essential Reading

10.2 Video Resources

10.3 Tools & Documentation

10.4 Related Projects in This Series

11. Self-Assessment Checklist

11.1 Understanding

11.2 Implementation

11.3 Growth

12. Submission / Completion Criteria