Project 9: Consentful UI Macro Runner

Build a deterministic, user-authored macro runner for your Sandbox Target with semantic steps, postconditions, visible control, and fail-closed safety.

Quick Reference

Main language: Kotlin
Alternative language: Java
Difficulty: Level 4 - Expert
Estimated time: 30-42 hours
Primary APIs: AccessibilityService, AccessibilityNodeInfo, optional dispatchGesture
Supporting tools: Room, Jetpack Compose, foreground-visible run controls
Main book: Release It!, Second Edition by Michael T. Nygard
Safety boundary: deterministic sandbox macros only; no sensitive, financial, permission, or security flows

1. Learning Objectives

You will be able to:

Model a UI macro as a persisted deterministic state machine.
Give every step a selector, precondition, action, postcondition, timeout, and retry policy.
Reacquire semantic targets from current UI state before acting.
Reject zero-match and multi-match selectors rather than guessing.
Distinguish action acceptance from observable completion.
Use postconditions as evidence of progress.
Handle process death and ambiguous external effects safely.
Design bounded retries that do not multiply effects.
Stop on package escape, revocation, prohibited controls, and timeout.
Provide Pause and Emergency Stop controls throughout execution.
Explain why coordinate-only actions are fragile and disabled by default.
Align deterministic user-authored automation with current Play policy limits.

Mastery signal: A sandbox form macro either proves each intended transition or stops with a precise, recoverable reason; it never guesses its way forward.

2. Theoretical Foundation

2.1 A Macro Is a State Machine, Not a Script of Taps

A brittle macro says “tap here, wait two seconds, tap there.” A reliable macro says “when the unique email field is visible and empty, set synthetic text; then wait until the field exposes the expected non-sensitive state.”

Ready -> Locating -> Preconditions -> Acting -> Verifying
  ^                                              |
  |                                              v
Paused <- Recoverable failure <- Retry decision  Completed
  |
  +-----------------------> Stopped / Failed

Each transition is explicit, persisted, observable, and bounded.

2.2 Step Contract

Every step needs:

Selector: semantic evidence for the intended control.
Precondition: facts that must hold before action.
Action: one bounded operation.
Postcondition: observable evidence of intended completion.
Timeout: maximum wait for each phase.
Retry policy: which failures may be attempted again.
Safety class: whether the step is allowed at all.

Without a postcondition, the runner knows only that it requested an action, not that the external UI accepted it.

2.3 External UI Is an Unreliable Dependency

The target app can redraw, navigate, show a keyboard, display a dialog, lose focus, change locale, update versions, or terminate. Accessibility events can be noisy or missing. Treat the UI as a remote system with eventual, partial observations.

The runner should repeatedly reacquire current semantic state within a deadline. It should not keep a node reference across waits.

2.4 Exactly-Once Is Usually Unavailable

Suppose the runner activates Submit, the target commits the form, and your process dies before persisting success. On restart, you cannot safely assume either failure or success.

Correct recovery:

mark the step outcome_unknown;
reacquire current UI;
query the postcondition;
if proven, record completion;
if disproven and action is retry-safe, retry within policy;
otherwise stop for user resolution.

This is the same ambiguity found in network payments and distributed job execution.

2.5 Idempotency and Retry Safety

Actions have different retry classes:

Read-only: safe to repeat.
Set-to-value: often retry-safe if postcondition is observable.
Toggle: unsafe without first observing current state.
Append: may duplicate content.
Submit: may produce external effects and become ambiguous.
Destructive or financial: prohibited in this project.

Retry policy belongs to the action contract, never to a generic catch-all loop.

2.6 Selector Safety

Before every action:

confirm current foreground package is allowlisted;
reacquire current window root;
apply the selector;
require exactly one match;
verify node is not password, secure, or prohibited;
check precondition;
act immediately or restart acquisition if generation changes.

A confidence score does not authorize acting on ambiguity.

2.7 Time and Waiting

Fixed sleeps are weak evidence. Prefer condition-based waits with monotonic deadlines.

wait until postcondition true
  wake on relevant event or short bounded poll
  reacquire current state
  cancel if package/window changes
  stop when monotonic deadline expires

Wall-clock changes must not extend a safety timeout.

2.8 Visible Control and Emergency Stop

High-authority automation requires immediate user control. The run controller should show routine, step, elapsed time, current phase, Pause, and Emergency Stop. Stop must cancel pending waits, prevent new actions, clear transient references, and persist a final reason.

An overlay must remain clearly owned by the app and must not imitate system surfaces.

2.9 Safety Policy

The reference runner permits only the owned Sandbox Target and fictional input. It prohibits:

passwords, PINs, OTPs, CAPTCHAs, and biometrics;
permission grants and accessibility enablement;
restricted-setting confirmation;
purchases, transfers, contracts, or destructive confirmation;
security dialogs or device-admin enrollment;
autonomous goal selection or open-ended planning.

2.10 Play Policy Context

Current Google Play policy distinguishes deterministic, human-defined automation from an app that autonomously initiates, plans, and executes actions. Apps using Accessibility API outside assistive-tool purposes face declaration, disclosure, consent, and permitted-use review. Build policy constraints into the product model before distribution.

2.11 Common Misconceptions

Misconception: A successful accessibility action means the task completed.

Correction: completion requires a separate observable postcondition.

Misconception: Retrying every exception improves reliability.

Correction: blind retries can duplicate or amplify external effects.

Misconception: A two-second delay makes the UI stable.

Correction: device load, animation, network state, and app design vary.

Misconception: Process death can restart the current step from the beginning.

Correction: externally committed actions may have ambiguous outcomes.

Misconception: Coordinates are a universal fallback.

Correction: rotation, density, keyboard, insets, and layout changes invalidate them.

2.12 Key Terms

Precondition: fact required before an action is safe and meaningful.
Postcondition: observable evidence that the intended effect occurred.
Ambiguous outcome: effect may have happened but cannot yet be proven.
Retry class: policy describing whether and how an action may repeat.
Monotonic deadline: timeout based on elapsed time unaffected by wall-clock changes.
Package escape: foreground focus leaves the explicitly allowed application.
Emergency stop: immediate user command preventing further actions.
Deterministic macro: explicit user-authored sequence without autonomous planning.

3. Project Specification

3.1 What You Will Build

Extend the Project 8 Sandbox Target and create:

a macro recorder that captures semantic intent, not raw coordinates;
an editor for selector, conditions, action, timeout, and retry policy;
a simulator that evaluates steps without acting;
a runner with visible step-by-step state;
Pause, Resume, and Emergency Stop controls;
a redacted run journal;
a recovery screen for ambiguous outcomes.

The reference macro fills fictional, non-sensitive form data and waits for a sandbox success state.

3.2 Functional Requirements

Run only against the explicit Sandbox Target allowlist.
Persist a versioned macro and step schema.
Require a semantic selector for each action step.
Require precondition and postcondition definitions.
Require exactly one current selector match.
Reject password and prohibited nodes.
Show current step, phase, elapsed time, and last observation.
Support Pause and Emergency Stop at any time.
Stop when the package changes unexpectedly.
Stop when accessibility access is disconnected or revoked.
Persist checkpoints before and after actions.
Recover process death as proven-complete, retry-safe, or ambiguous.
Keep coordinate-only actions off by default and visibly fragile.

3.3 Non-Functional Requirements

No real credentials or personal data in macros, traces, or tests.
All waits and retries are bounded.
The action executor accepts immutable step plans.
Framework nodes remain transient.
The runner cannot synthesize new goals or steps.
Logs are redacted and exportable for review.
Stop latency is measured and documented.
The UI remains accessible while automation is active.

3.4 Reference Macro

Macro: Submit sandbox contact form

Step 1
  selector: package + resource_id=email + editable_role
  precondition: exactly one match; not password; field empty
  action: set synthetic value "learner@example.test"
  postcondition: field reports non-empty synthetic-value marker
  timeout: 3 seconds
  retry: set-to-value, at most once

Step 2
  selector: package + resource_id=submit + button_role
  precondition: exactly one enabled match; validation clear
  action: activate
  postcondition: success semantic marker visible
  timeout: 8 seconds
  retry: no blind retry; query postcondition after ambiguity

This is a design transcript, not runnable implementation.

3.5 Example Run

Run 42: Submit sandbox contact form
Step 1/2: set synthetic email
  locate: unique
  precondition: passed
  action request: accepted
  postcondition: passed

Step 2/2: submit
  locate: unique
  precondition: passed
  action request: accepted
  postcondition: passed

Result: completed
Retained sensitive text: none

3.6 Real-World Outcome

A reviewer records, edits, simulates, and runs the sandbox macro. They can pause or stop it. Duplicate controls, password fields, package escape, revocation, and timeout all fail closed. Killing the process immediately after Submit produces an ambiguous recovery state that queries the postcondition before doing anything else.

4. Solution Architecture

4.1 High-Level Architecture

 macro editor ---> versioned macro repository
                           |
                      compiler/validator
                           |
                    immutable run plan
                           |
                    persisted run state
                           |
 +-------------------------+--------------------------+
 | state machine                                      |
 | Locate -> Precheck -> Act -> Verify -> Checkpoint  |
 +-------------------------+--------------------------+
                           |
             accessibility adapter + package gate
                           |
                    Sandbox Target UI
                           |
              relevant events / current snapshots
                           |
                 postcondition evaluator
                           |
             redacted journal + visible controller

4.2 Component Responsibilities

Component	Responsibility	Must Not Do
Macro repository	store versioned user-authored definitions	store node objects
Compiler	validate steps and create immutable plan	invent missing actions
Safety policy	reject prohibited targets and actions	be bypassed by retry logic
Run state machine	own legal transitions and checkpoints	hide ambiguous outcomes
Selector resolver	reacquire exactly one safe current target	choose the nearest match
Action adapter	issue one bounded Android action	decide overall workflow
Postcondition evaluator	prove external state	trust action return alone
Controller	expose progress, Pause, Stop	conceal active execution
Journal	record redacted causation and outcomes	retain field contents

4.3 Macro Schema

MacroDefinition {
  schemaVersion
  macroId
  targetPackage
  steps[]
  createdByUser
}

StepDefinition {
  stepId
  selector
  precondition
  action
  postcondition
  phaseTimeouts
  retryClass
  safetyClass
}

RunCheckpoint {
  runId
  stepIndex
  phase
  attempt
  planVersion
  lastObservationDigest
  outcome
}

4.4 State Machine

Created
  |
  v
Validating --invalid--> Rejected
  |
  v
Locating --0/many--> FailedSafe
  |
  v
Prechecking --false--> Blocked
  |
checkpoint before action
  |
  v
Acting --process death--> OutcomeUnknown
  |
  v
Verifying --timeout--> RetryDecision
  |                         |
  |                         +--> Locate again, if safe
  |                         +--> UserResolution
  v
StepComplete --> next step --> RunComplete

Any active state --Pause--> Paused
Any active state --Emergency Stop/package escape/revocation--> Stopped

4.5 Execution Algorithm

load immutable run plan
for each step:
  assert run remains active and package allowed
  persist locating checkpoint
  reacquire current root and resolve selector
  require exactly one non-prohibited match
  evaluate precondition
  persist before-action checkpoint
  issue one bounded action
  discard node reference
  wait for postcondition using current-state reacquisition
  if proven: persist step-complete checkpoint
  else: apply action-specific retry decision
finish only when every postcondition is proven

4.6 Recovery Algorithm

on process restart:
  load last durable checkpoint
  revalidate capability, target package, macro version, and safety policy
  if checkpoint is before action: locate and continue
  if checkpoint is after proven postcondition: advance
  if action may have occurred but proof was not saved:
      mark OutcomeUnknown
      query current postcondition before considering retry
  if state cannot be proven safely: require user resolution

4.7 Critical Invariants

The runner never acts outside the allowlisted package.
Every action requires one unique current safe target.
Every step has a precondition and postcondition.
Fixed sleeps are not completion evidence.
Retry policy is action-specific and bounded.
Process death never implies automatic blind replay.
Emergency Stop prevents all subsequent actions.
The runner never generates new goals or steps.

5. Implementation Guide

5.1 Environment Setup

Complete Project 8 first.
Use a dedicated emulator with only fictional sandbox data.
Add deterministic success, delay, duplicate, and failure modes to the Sandbox Target.
Use SDK Platform 36 for the reference track.
Prepare process-kill and accessibility-revocation test commands.
Keep network access out of the reference macro.

5.2 Suggested Structure

ui-macro-runner/
  macros/schema/
  macros/editor/
  compiler/
  safety/
  runner/state-machine/
  runner/checkpoints/
  accessibility/selectors/
  accessibility/actions/
  conditions/
  controller/
  journal/
  sandbox-target/
  tests/

5.3 Core Question

How can an automation act on a changing external UI while proving that each intended step completed and stopping safely when proof is unavailable?

5.4 Concepts to Understand First

Persisted finite-state machines.
Selector uniqueness and freshness.
Preconditions and postconditions.
Monotonic timeouts and condition waits.
Idempotency and ambiguous effects.
Accessibility action and gesture constraints.
Policy-bounded deterministic automation.

5.5 Design Questions

Which step fields are mandatory at save time?
Which actions are retry-safe and why?
Where is the checkpoint relative to action invocation?
How is outcome ambiguity represented?
What cancels in-flight waits?
How quickly must Emergency Stop take effect?
Which target changes require immediate failure?
What evidence can be recorded without sensitive content?

5.6 Thinking Exercise

Trace durable state and safe next action for:

process death before action;
process death after action but before postcondition;
process death after postcondition but before checkpoint;
selector changes from one match to two;
target package leaves foreground during a wait;
service access is revoked during action dispatch;
user presses Stop while retry delay is pending.

5.7 Interview Questions

Why is action acceptance not proof of completion?
How do you recover an ambiguous external effect?
When is retrying a UI action unsafe?
Why should a selector be reacquired immediately before acting?
How do monotonic deadlines improve safety?
What separates deterministic automation from autonomous planning?

5.8 Hints in Layers

Hint 1 - Start with simulation

Evaluate selectors and conditions without executing actions.

Hint 2 - Persist state transitions

Make recovery correct before adding multiple steps.

Hint 3 - Prove one set-to-value step

Choose an idempotent sandbox action with an observable postcondition.

Hint 4 - Add ambiguous Submit

Kill the process around the action boundary and inspect recovery.

Hint 5 - Add safety stops last but test them everywhere

Stop, package escape, and revocation should interrupt every active phase.

5.9 Books That Will Help

Topic	Book	Suggested focus
Reliability	Release It!	timeouts, retries, stability patterns
Distributed effects	Designing Data-Intensive Applications	delivery and idempotency
State modeling	Domain Modeling Made Functional	explicit states and transitions
Security	Foundations of Information Security	least privilege and fail-closed design

5.10 Implementation Phases

Phase 1 - Schema and compiler

Define versioned macro and step schemas.
Require all safety fields.
Reject unsupported selector types.
Produce an immutable run plan.

Phase 2 - Simulator

Resolve selectors without acting.
Evaluate preconditions.
Show match counts and prohibited reasons.
Save safe fixtures.

Phase 3 - Persisted state machine

Define legal transitions.
Persist checkpoints transactionally.
Add Pause and Stop states.
Test reducer exhaustively.

Phase 4 - First safe action

Implement one set-to-value sandbox step.
Reacquire immediately before action.
Verify a semantic postcondition.
Discard transient node references.

Phase 5 - Multi-step macro

Add navigation and Submit.
Add condition-based waits.
Add per-phase deadlines.
Add action-specific retry decisions.

Phase 6 - Visible controller

Show macro, step, phase, and elapsed time.
Add Pause, Resume, and Emergency Stop.
Make observation and execution conspicuous.
Measure stop latency.

Phase 7 - Recovery

Kill the process at every checkpoint boundary.
Query postconditions after ambiguity.
Require user resolution when proof is impossible.
Revalidate macro version and safety policy.

Phase 8 - Hardening

Test package escape and revocation.
Test duplicate and missing targets.
Prohibit password and unsafe actions.
Audit traces for synthetic secrets.

5.11 Key Decisions

Semantic selectors before coordinate fallback.
Exactly one match before action.
Postconditions as completion proof.
Monotonic bounded waits.
Action-specific retry policy.
Persisted ambiguity rather than inferred success.
Immediate visible user control.

6. Testing Strategy

6.1 State-Machine Tests

Every legal transition succeeds.
Every illegal transition is rejected.
Pause and Stop are reachable from all active phases.
Stop is terminal for action dispatch.
Recovery selects the correct path for each checkpoint.
Macro-version mismatch blocks continuation.

6.2 Selector and Condition Tests

Zero matches fail safely.
Two matches fail as ambiguous.
Password target is prohibited.
Stale selector reacquires current state.
Precondition false blocks action.
Postcondition false until deadline times out.

6.3 Integration Tests

Run the complete sandbox form macro.
Rotate during locate and verify.
Show and hide the keyboard.
Delay the success state.
Change locale between recording and execution.
Leave the target package mid-step.
Revoke accessibility during a wait.
Press Emergency Stop during every active phase.

6.4 Process-Death Matrix

Kill point	Expected recovery
before locate	restart locate
after unique locate	reacquire; never reuse node
before action checkpoint	continue from checkpoint
after action request	query postcondition; mark ambiguous
after postcondition proven	verify checkpoint then advance
during retry wait	restore deadline or stop if expired
after Stop	remain stopped; no further actions

6.5 Safety Tests

Attempt password-field selection.
Attempt coordinate-only step with default policy.
Attempt permission-dialog targeting.
Attempt unknown package macro.
Attempt unbounded retry configuration.
Attempt macro with missing postcondition.
Verify all are rejected before execution.

6.6 Privacy and Observability Tests

Seed a unique synthetic secret.
Search Room, logs, exports, and crash reports.
Confirm journal records selector evidence, not field contents.
Confirm correlation from macro to step to action.
Confirm Emergency Stop reason is durable.

6.7 Acceptance Transcript

Provide runs for success, ambiguous process death, duplicate match, package escape, timeout, revocation, password rejection, Pause/Resume, and Emergency Stop. Each transcript must show the final typed outcome and no private text.

7. Common Pitfalls & Debugging

Problem 1: Macro races ahead after a tap

Why: action return was mistaken for completed UI transition.
Fix: wait for a semantic postcondition with a deadline.
Quick test: delay the Sandbox Target success state.

Problem 2: Step hits the wrong duplicate control

Why: resolver chose a first or nearest match.
Fix: require exactly one match and stop on ambiguity.
Quick test: duplicate the target row.

Problem 3: Restart repeats Submit

Why: recovery treated an ambiguous effect as failure.
Fix: query postcondition before any retry.
Quick test: kill immediately after action request.

Problem 4: Timeout changes when clock changes

Why: wall time was used for elapsed safety deadlines.
Fix: use monotonic elapsed time.
Quick test: change wall clock during a wait.

Problem 5: Stop button responds but action continues

Why: cancellation is checked only between steps.
Fix: make waits and adapters cancellation-aware.
Quick test: stop during locate, act, verify, and retry wait.

Problem 6: Macro continues in another app

Why: package scope is checked only at run start.
Fix: validate package before every observation and action.
Quick test: navigate away during postcondition wait.

8. Extensions & Challenges

Extension 1: Macro Linter

Detect missing postconditions, fragile selectors, toggle retries, excessive timeouts, and prohibited steps.

Extension 2: Dry-Run Timeline

Simulate each step and explain what evidence would be required without performing actions.

Extension 3: Version Drift Report

Compare selector results across two Sandbox Target versions.

Extension 4: Human Checkpoint

Add an explicit user confirmation step for a harmless sandbox transition, without automating the confirmation itself.

Extension 5: Property-Based State Testing

Generate legal event sequences and prove Stop is terminal and ambiguity never auto-retries unsafe actions.

Forbidden Extensions

Credential, OTP, CAPTCHA, biometric, or security-flow automation.
Purchases, banking, legal acceptance, or destructive confirmation.
Permission or accessibility-setting automation.
Autonomous generation of goals or action plans.
Unrestricted third-party package operation.

9. Real-World Connections

Reliable UI automation is used in testing, assistive technology, repetitive-work tools, and controlled enterprise workflows. Mature products combine semantic selectors, app-specific adapters, schema migration, policy review, compatibility labs, supportable traces, and explicit user scope.

The process-death lesson transfers directly to payment APIs, queues, workflow engines, and robotics: once an external effect may have happened, recovery must observe before repeating.

Production Gap Analysis

Learning build	Production requirement
owned sandbox	app drift and third-party compatibility
simple macros	schema migration and editor validation
local journal	retention, user export, support consent
one-device tests	OEM, density, locale, and version matrix
manual policy notes	Play declaration and ongoing compliance

10. Resources

Official Android Documentation

Official Policy

11. Self-Assessment Checklist

Knowledge

I can explain preconditions versus postconditions.
I understand ambiguous external effects.
I can classify action retry safety.
I can explain monotonic deadlines.
I understand deterministic automation policy boundaries.

Design

Every step has all seven required contract fields.
Every action requires one unique safe current match.
Run state and checkpoints are durable.
Emergency Stop is terminal.
The runner cannot invent steps.

Verification

I tested every process-death boundary.
I tested zero and multiple selector matches.
I tested package escape and revocation.
I tested Pause, Resume, and Stop in every phase.
I scanned all artifacts for synthetic secrets.

Reflection

I can defend every retry policy.
I can describe one ambiguous run without guessing.
I documented why coordinate fallback is disabled.
I can name all prohibited automation classes.

12. Submission / Completion Criteria

Submit:

Versioned macro and step schema.
Full persisted state-machine diagram.
Safety-policy table with prohibited actions.
Selector, precondition, and postcondition examples.
Retry-class decision record.
Process-death recovery matrix.
Emergency Stop latency measurement.
Complete automated test report.
Redacted acceptance transcripts.
Play policy assessment for the deterministic scope.

Definition of Done

Expanded from ANDROID_AUTOMATION_APPS_MASTERY.md. For the complete learning path, see the parent guide.