Project 8: Accessibility Inspector

Build a consentful, package-allowlisted inspector that explains Android accessibility events, semantic UI trees, and selector quality without retaining sensitive content.

Quick Reference

Main language: Kotlin
Alternative language: Java
Difficulty: Level 3 - Advanced
Estimated time: 24-34 hours
Primary APIs: AccessibilityService, AccessibilityServiceInfo, AccessibilityNodeInfo
Supporting tools: Jetpack Compose, local overlay UI, a separate Sandbox Target app
Main book: Designing with the Mind in Mind by Jeff Johnson
Safety boundary: inspect only explicit test packages; never inspect password or secure content

1. Learning Objectives

After this project, you can:

Explain the user-controlled lifecycle of an accessibility service.
Distinguish event metadata, window state, and a point-in-time semantic tree.
Traverse nodes without retaining framework objects beyond their safe lifetime.
Restrict observation to an explicit package allowlist.
Detect and exclude password, secure, and prohibited targets.
Rank resource ID, role, description, text, structural, and coordinate selectors.
Explain selector confidence and fragility to a user.
Reacquire a node from current UI state rather than caching a stale reference.
Bound event volume, traversal depth, and overlay updates.
Freeze immediately when focus leaves the allowed package.
Create privacy-safe diagnostics and test fixtures.
Relate technical design to current Google Play Accessibility API policy.

Mastery signal: The inspector identifies safe semantic controls in a sandbox, explains selector quality, and produces no observation when the foreground package is outside scope.

2. Theoretical Foundation

2.1 Accessibility Is a High-Authority User-Enabled Service

An AccessibilityService exists primarily to assist users with disabilities. It receives system-mediated events and, when configured, can inspect interactive windows. The user enables it in system Settings; an app cannot self-enable it or automate the consent journey.

The service declaration, current Settings grant, runtime connection, configured event scope, and current foreground package are separate facts. Your UI must not collapse them into one enabled boolean.

2.2 Events Are Hints, Not Complete State

Accessibility events report changes: focus moved, content changed, a window changed, or a view was clicked. They may be duplicated, coalesced, delayed, or incomplete. Treat them as prompts to inspect current state, not as an authoritative event-sourced database.

event arrives
    |
validate service + package scope
    |
reacquire current root
    |
build bounded semantic snapshot
    |
rank safe selector candidates

2.3 Semantic Trees

AccessibilityNodeInfo exposes accessibility-oriented properties such as class, role-like behavior, bounds, resource view ID when available, text, content description, actions, focusability, and hierarchy. It is not a stable DOM. Nodes can become stale whenever the external application redraws or replaces its window.

Never make the node itself part of the durable domain model. Extract minimal, redacted features and recycle or release framework references according to current API behavior.

2.4 Selector Hierarchy

A selector describes how to reacquire an intended control. Candidate strength usually follows this rough order:

Stable resource ID scoped to package.
Stable semantic role/class plus unique accessible label.
Content description with role and ancestry constraints.
Bounded visible text with localization warning.
Structural path with stable anchors.
Coordinates as a fragile, opt-in fallback.

No single signal is universally best. Resource IDs can change; text localizes; structure shifts; duplicate labels create ambiguity; coordinates break with density, rotation, keyboard, and animation.

2.5 Confidence Is Explainable Evidence

Do not output a magical score. Show the contributing facts:

candidate: package + resource_id + role
uniqueness: 1 match in current window
stability evidence: ID present across 5 snapshots
penalties: dynamic list ancestor
verdict: high confidence, retest after app update

Confidence must never convert multiple matches into permission to guess.

2.6 Privacy Boundary

Accessibility data can expose everything visible to the user. Your inspector should:

observe only a separate sandbox target selected by the user;
stop outside the allowlist;
reject password nodes and descendants;
avoid text persistence by default;
bound transient text used for selector analysis;
keep all processing on device;
provide a persistent stop control;
make active observation conspicuous.

2.7 Event Storms and Traversal Budgets

Animations, lists, typing, and layout changes can produce rapid events. Each event must not trigger an unbounded full-tree traversal. Define:

relevant event types;
debounce or coalescing window;
maximum traversal depth;
maximum node count;
maximum label length;
cancellation when the active window changes;
overlay refresh rate.

2.8 Overlays and Secure UX

An inspector may highlight bounds using an accessibility overlay. The overlay must not imitate system dialogs, intercept unrelated input, conceal application context, or operate outside the allowlist. Keep controls clear, dismissible, and visibly owned by your app.

2.9 Play Policy Context

Google Play requires appropriate Accessibility API use, declaration, prominent disclosure, and consent when an app is not an accessibility tool. Deterministic user-authored automation can be permitted, but autonomous initiation, planning, and execution are restricted. Policy compliance is a product constraint, not merely a manifest detail.

2.10 Common Misconceptions

Misconception: Accessibility nodes are stable object references.

Correction: reacquire from current state and treat snapshots as ephemeral.

Misconception: Visible text is a reliable selector.

Correction: it localizes, duplicates, truncates, and changes with data.

Misconception: A confidence score permits selecting one of several matches.

Correction: ambiguity should stop selection.

Misconception: Package filters in metadata eliminate all privacy risk.

Correction: runtime package validation and immediate freeze are still required.

Misconception: Custom-drawn controls always have hidden semantic nodes.

Correction: inaccessible canvases may expose no actionable structure.

2.11 Key Terms

Accessibility event: notification that accessible UI state may have changed.
Semantic tree: accessibility-oriented representation of current UI.
Node snapshot: bounded immutable projection of a transient framework node.
Selector: declarative evidence used to reacquire an intended control.
Ambiguity: more than one control satisfies required selector evidence.
Staleness: referenced UI evidence no longer describes current state.
Package allowlist: explicit set of applications the user permits inspection within.
Accessibility overlay: service-owned window used for assistive presentation.

3. Project Specification

3.1 What You Will Build

Create two apps or modules:

Sandbox Target

a form with labels and editable fields;
a scrolling list with repeated row labels;
localized text resources;
a button with a stable resource ID;
a button with description but no useful text;
a password field;
a custom-drawn control with poor semantics;
a page that replaces its tree after a delay.

Accessibility Inspector

capability and connection dashboard;
explicit sandbox package enrollment;
current package, window, and event viewer;
bounded semantic tree browser;
selected-node highlighter;
selector candidate ranking with explanations;
stop and revoke guidance;
privacy-safe export of structural evidence.

3.2 Functional Requirements

Observe only explicitly allowlisted packages.
Freeze and clear transient state when focus leaves scope.
Show current package, window ID, event kind, and snapshot generation.
Traverse within node-count and depth budgets.
Prohibit password and secure targets.
Show role/class, bounds, actions, and redacted label presence.
Produce ranked selector candidates.
Display match count and rejection reason for every candidate.
Explain coordinate fallback as fragile and keep it off by default.
Reacquire a selection from current root before highlighting.
Expose a persistent Stop Inspection control.
Handle service revocation without deleting configuration.

3.3 Non-Functional Requirements

No raw screen text in Room, logs, analytics, or exports.
No network permission is needed for the reference implementation.
Event processing remains bounded under animation and typing.
Overlay must not block target-app navigation.
The service state is truthful after process death.
The inspector is testable with synthetic semantic fixtures.
Policy disclosure is written in plain language.

3.4 Example Interaction

Inspection active
  package: dev.example.sandbox
  window: application
  event: window_content_changed
  snapshot: 118
  nodes visited: 42 / 200 budget

Selected control: Submit order
Candidate A: resource_id + role
  matches: 1
  confidence: high
Candidate B: text + ancestry
  matches: 2
  confidence: rejected_ambiguous
Coordinate fallback: disabled

Outside the allowlist:

Inspection frozen
Reason: foreground_package_not_allowed
Retained node text: none

3.5 Real-World Outcome

A reviewer explores the sandbox in two locales, sees semantic nodes, and compares selector candidates. Duplicate labels are rejected as ambiguous. The password field is prohibited. The custom-drawn control is honestly reported as lacking semantics. Switching to another app immediately freezes the inspector and clears transient evidence.

4. Solution Architecture

4.1 High-Level Architecture

 Android accessibility framework
              |
       AccessibilityService
              |
 +------------+-------------+
 | capability/connection    |
 | foreground package gate  |
 +------------+-------------+
              |
       event coalescer
              |
       current-root reader
              |
   bounded semantic projector
              |
     prohibited-node filter
              |
       selector ranker
          /         \
 inspector UI     safe export
      |
 accessibility overlay

4.2 Component Responsibilities

Component	Responsibility	Must Not Do
Service adapter	receive lifecycle and event callbacks	store nodes globally
Package gate	validate current package every cycle	infer consent from installation
Coalescer	bound redundant event work	hide a package transition
Projector	extract minimal immutable features	retain raw framework nodes
Prohibited filter	exclude password and unsafe targets	permit override for convenience
Selector ranker	compare evidence and match counts	choose among ambiguous matches
Overlay	highlight verified bounds	imitate system UI
Safe exporter	emit structural diagnostics	include raw visible text

4.3 Data Structures

SemanticSnapshot {
  generation
  packageName
  windowId
  capturedAtElapsed
  nodes[]
  truncated
}

NodeFeature {
  localId
  parentLocalId?
  classBucket
  resourceId?
  labelFingerprint?
  boundsBucket
  actions[]
  password
}

SelectorCandidate {
  predicates[]
  currentMatchCount
  evidence[]
  penalties[]
  verdict
}

These schemas are conceptual and intentionally omit runnable code.

4.4 Traversal Algorithm

when relevant event survives coalescing:
  verify service is connected
  verify foreground package is allowlisted
  acquire current active-window root
  breadth-first traverse within depth and node budgets
  reject prohibited nodes before optional label analysis
  project minimal features
  release framework references
  publish snapshot only if package and window generation still match

4.5 Selector Ranking Algorithm

for selected safe node:
  create candidates from strongest available evidence
  reacquire current root
  evaluate each candidate across current window
  reject candidate when match count is not exactly one
  score remaining evidence with explicit bonuses and penalties
  show explanation and fragility warning
  never persist raw node text

4.6 Critical Invariants

No package outside the allowlist yields a snapshot.
Password nodes never become selector targets.
Framework node objects do not cross the adapter boundary.
Every highlight uses a freshly reacquired unique match.
Ambiguity stops selection.
Coordinate-only selection is off by default.
Exports contain structure and decisions, not private content.

5. Implementation Guide

5.1 Environment Setup

Use current stable Android Studio and SDK Platform 36.
Create a dedicated Android 13+ emulator with no personal accounts.
Build the Sandbox Target first.
Use fictional data and synthetic labels only.
Keep the inspector offline for the reference build.
Prepare two locales and two display sizes.

5.2 Suggested Structure

accessibility-inspector/
  service/
  capability/
  scope/
  events/
  snapshot/
  selectors/
  overlay/
  privacy/
  ui/
  sandbox-target/
  tests/fixtures/

5.3 Core Question

How can the app identify the intended control from stable semantic evidence while observing the smallest possible amount of UI data?

5.4 Concepts to Understand First

Accessibility service configuration and lifecycle.
Event types and interactive-window retrieval.
Transient node ownership and stale references.
Breadth-first bounded traversal.
Selector specificity, uniqueness, and stability.
Accessibility overlay behavior.
Play policy disclosure and deterministic automation limits.

5.5 Design Questions

Which event types actually require a new snapshot?
How will a package change cancel in-flight traversal?
Which node fields are safe to project?
How is a label compared without retaining it?
What evidence makes a selector unique now?
What evidence suggests it will remain stable later?
How does the user know observation is active?
How quickly can they stop it?

5.6 Thinking Exercise

For each sandbox control, predict selector behavior after:

locale changes;
list reordering;
duplicate labels;
app redesign with new resource IDs;
keyboard appearance;
display rotation;
delayed tree replacement;
focus leaving the allowlist.

Write which selector survives, which fails, and whether failure is safe.

5.7 Interview Questions

Why are accessibility events insufficient as complete state?
Why should nodes be reacquired before action or highlight?
How do you rank resource ID versus visible text?
What is the correct response to two selector matches?
How do you prevent event storms from exhausting the app?
What Play policy constraints shape this design?

5.8 Hints in Layers

Hint 1 - Make the sandbox deterministic

Own the target UI so failures are explainable.

Hint 2 - Enforce scope before traversal

Package validation belongs before optional node inspection.

Hint 3 - Snapshot minimal features

Keep IDs, role buckets, bounds buckets, actions, and redacted label evidence.

Hint 4 - Rank by uniqueness

A strong-looking selector with two matches is unusable.

Hint 5 - Test cancellation

Switch packages during a large traversal and verify no stale snapshot publishes.

5.9 Books That Will Help

Topic	Book	Suggested focus
Human factors	Designing with the Mind in Mind	perception and feedback
Accessibility	Accessibility for Everyone	inclusive interaction principles
Security	Foundations of Information Security	least privilege and privacy
Reliability	Release It!	bounds, cancellation, overload

5.10 Implementation Phases

Phase 1 - Sandbox Target

Build stable and unstable controls.
Add duplicate and localized labels.
Add password and custom-drawn cases.
Add delayed tree replacement.

Phase 2 - Capability state

Declare service purpose and event scope.
Add plain-language disclosure.
Show grant and connection separately.
Add a persistent Stop control.

Phase 3 - Package scope

Enroll only the sandbox package.
Validate foreground package on every cycle.
Cancel and clear when scope changes.
Test immediate freeze.

Phase 4 - Bounded snapshots

Filter relevant event types.
Coalesce bursts.
Traverse within budgets.
Project immutable safe features.

Phase 5 - Inspector UI

Display event and snapshot metadata.
Render tree structure without raw private text.
Add current-node detail.
Expose truncation and freshness.

Phase 6 - Selector ranker

Generate semantic candidates.
Re-evaluate against the current tree.
Require exactly one match.
Explain evidence and penalties.

Phase 7 - Overlay and hardening

Reacquire before highlight.
Bound overlay refresh.
Block password and unsafe nodes.
Add safe diagnostic export.

5.11 Key Decisions

Explicit allowlist, never broad observation.
Current-tree lookup, never durable node references.
Explainable selector evidence, never opaque confidence.
Ambiguity as failure, never guesswork.
Offline processing and minimal retention.
Coordinate fallback disabled by default.

6. Testing Strategy

6.1 Unit Tests

Package gate denies unknown and empty package names.
Traversal obeys depth and node budgets.
Password nodes are excluded.
Duplicate selector matches are rejected.
Resource ID candidate outranks coordinate fallback.
Localization penalty is applied to text selectors.
Stale generation cannot publish.
Safe export omits seeded label markers.

6.2 Instrumentation Tests

Inspect every Sandbox Target control.
Change locale and rerun selector ranking.
Reorder the repeated list.
Replace the tree after selection.
Switch to an unselected app mid-traversal.
Revoke service access while overlay is visible.
Kill and recreate the inspector process.

6.3 Volume Tests

Generate rapid text-change events.
Animate a list while inspection is active.
Confirm coalescing and cancellation metrics.
Confirm UI remains responsive.
Confirm memory returns to baseline after snapshots expire.

6.4 Privacy Tests

Seed unique secrets in safe and password fields.
Search databases, logs, exports, and crash artifacts.
Verify no snapshot outside allowlist.
Verify Stop clears transient data.
Verify network traffic is absent in the reference build.

6.5 Failure Matrix

Failure	Expected behavior
service disconnected	inspector inactive; configuration retained
package changes	traversal canceled; transient view cleared
root unavailable	typed unavailable snapshot
selector matches zero	stale or changed target
selector matches many	ambiguous; no highlight
password node	prohibited
custom canvas	unsupported semantics reported
budget exceeded	truncated snapshot with visible reason

6.6 Acceptance Evidence

Capture redacted evidence for stable ID, localized text, duplicate label, password, custom canvas, stale tree, event storm, package escape, Stop, and revocation. Include match counts and selector explanations.

7. Common Pitfalls & Debugging

Problem 1: Old node fails after redraw

Why: a transient framework object was cached.
Fix: persist a selector candidate and reacquire current state.
Quick test: select, trigger delayed replacement, then highlight.

Problem 2: Inspector keeps updating in another app

Why: package scope was checked only during enrollment.
Fix: validate foreground package before every traversal and publication.
Quick test: switch apps mid-traversal.

Problem 3: CPU spikes while typing

Why: every text event causes a full traversal.
Fix: filter, coalesce, cancel stale work, and cap traversal.
Quick test: type continuously while profiling event rate.

Problem 4: Selector works only in English

Why: visible text was treated as stable identity.
Fix: prefer ID and role; mark text dependence explicitly.
Quick test: switch the Sandbox Target locale.

Problem 5: Overlay highlights wrong duplicate

Why: ranking ignored current match count.
Fix: require exactly one reacquired match.
Quick test: create duplicate row labels.

Problem 6: Secret appears in diagnostic export

Why: export serialized the UI model directly.
Fix: build a dedicated safe export schema.
Quick test: search for a seeded unique marker.

8. Extensions & Challenges

Extension 1: Selector Stability Lab

Run synthetic app revisions and compare candidate survival.

Extension 2: Accessibility Quality Audit

Report missing labels, tiny targets, duplicate descriptions, and inaccessible custom controls in your own sandbox.

Extension 3: Window Topology Viewer

Visualize application, system, keyboard, and overlay windows without inspecting prohibited content.

Extension 4: Recorded Safe Fixtures

Create hand-authored semantic fixtures that contain no real user data.

Extension 5: Policy Disclosure Prototype

Write onboarding, in-context disclosure, Settings guidance, and deletion explanations for review.

Forbidden Extensions

Password, PIN, OTP, CAPTCHA, or biometric inspection.
Permission-dialog or restricted-setting automation.
Banking, purchase, or security confirmation targeting.
Silent cross-app observation.
Server upload of raw trees or screenshots.

9. Real-World Connections

Semantic inspection underlies screen readers, switch access, testing tools, assistive input, and deterministic UI automation. Production systems must handle custom views, WebView semantics, Compose semantics, internationalization, OEM differences, app updates, policy review, and support diagnostics.

The project also teaches a general observability principle: inspect a changing external system through bounded snapshots, attach freshness, and never confuse a snapshot with a durable object model.

Production Gap Analysis

Learning build	Production requirement
owned sandbox	diverse third-party apps and update drift
local allowlist	reviewable user scope and migration
offline inspector	privacy threat model and retention controls
simple selector ranker	longitudinal stability evidence
manual disclosure	Play declaration and compliance operations

10. Resources

Official Android Documentation

Official Policy

11. Self-Assessment Checklist

Knowledge

I can distinguish events from current UI state.
I can explain node staleness.
I can rank selector evidence and penalties.
I understand traversal backpressure.
I can summarize current Play policy constraints.

Design

Observation is restricted to an explicit package allowlist.
Password and prohibited targets are blocked.
Framework nodes never enter durable storage.
Ambiguity prevents highlight or action.
Coordinate fallback is disabled by default.

Verification

I tested locale and duplicate labels.
I tested stale-tree reacquisition.
I tested package escape and revocation.
I tested event storms and traversal limits.
I scanned diagnostics for seeded secrets.

Reflection

I can defend every projected node field.
I can explain one unsupported custom-control case.
I documented why the inspector remains offline.
I can identify selector evidence that may drift after updates.

12. Submission / Completion Criteria

Submit:

Sandbox Target behavior inventory.
Service capability and connection state machine.
Package-scope and prohibited-target policy.
Bounded traversal algorithm and budgets.
Safe semantic snapshot schema.
Selector ranking rubric with examples.
Overlay threat review.
Unit, instrumentation, volume, and privacy test report.
Redacted acceptance transcript.
Play policy disclosure and declaration notes.

Definition of Done

Expanded from ANDROID_AUTOMATION_APPS_MASTERY.md. For the complete learning path, see the parent guide.