Project 12: PocketOps Mobile Edge Automation Node

Integrate the entire sprint into a signed, recoverable, security-gated mobile edge node that another operator can install, interrupt, upgrade, restore, and explain from evidence.

Quick Reference

Attribute Value
Difficulty Level 4 - Expert capstone
Time estimate 45-70 hours
Main language Python and shell orchestration with the P09 native C probe
Alternatives Go or Rust core with thin shell/Android adapters
Prerequisites P01-P11 or equivalent contracts and evidence
Primary concepts Integrated contracts, capability negotiation, event ledger, Android lifecycle recovery, signed release, authorization, observability, disaster recovery
Main tools Classic Termux and compatible add-ons, SQLite or equivalent private store, runit/termux-services, Termux bridges, SSH, signed APT repository
Observable outcome A clean-device 12-step acceptance run passes after permission denial, offline operation, process death, reboot, upgrade/rollback, tamper attempts, and restore

1. Learning Objectives

By completing this capstone, you will be able to:

  1. Integrate multiple entry surfaces through one versioned operation model rather than duplicate business logic.
  2. Design a durable event ledger with correlation, idempotency, leases, retry, dead-letter, and audit semantics.
  3. Keep Android-facing adapters thin and make capability denial an explicit degraded mode.
  4. Prove that accepted work survives worker death, supervisor death, Termux process-tree death, reboot, and network loss.
  5. Separate package-owned files, mutable private state, secrets, shared imports/exports, and recovery artifacts.
  6. Install and upgrade from a signed Termux repository while preserving state and rejecting unauthorized artifacts.
  7. Authorize Widget, GUI, Tasker, SSH, and network actions independently under least privilege.
  8. Build health around liveness, readiness, degradation, queue age, recovery, and Android constraints—not PID presence.
  9. Run a security audit as a release gate and close real findings without leaking evidence.
  10. Execute rollback, encrypted backup restore, and clean rebuild from documented trust roots.
  11. Explain where Termux remains the right product boundary and where a conventional Android APK is required.
  12. Produce a redacted evidence bundle another operator can reproduce without developer memory.

2. Theoretical Foundation

2.1 Core Concepts

The capstone is a system of contracts, not a pile of scripts. PocketOps has many entry surfaces: a shell, home-screen Widget, Termux:GUI activity, Tasker event, SSH forced command, boot event, supervised worker, and optional local network endpoint. Each surface translates an external request into the same typed operation. The core validates, persists, executes, and returns a typed result. Adapters may format or transport that result, but they do not own domain state.

  shell   Widget   GUI   Tasker   SSH   Boot   local webhook
    |       |       |      |      |      |          |
    +-------+-------+------+------+------+----------+
                            |
                  versioned operation envelope
                            |
                  +---------v----------+
                  | validation + policy|
                  +---------+----------+
                            |
                  durable event ledger
                  ready/leased/done/DLQ
                            |
                  +---------v----------+
                  | supervised workers |
                  +---------+----------+
                            |
        +-------------------+--------------------+
        |                   |                    |
  Android adapters    network adapters     private storage
  API/notification    sync/SSH/webhook     vault/config/DB
        |                   |                    |
        +-------------------+--------------------+
                            |
                 result + audit + health evidence

Termux is Android-hosted, not a general Linux server. The system runs as the Termux app UID on Android’s kernel and Bionic, with packages beneath PREFIX and mutable state beneath private HOME. Shared storage remains an import/export boundary, not an executable root. A successful terminal experiment does not prove Boot, Widget, Tasker, service, or SSH environments; production entry points establish explicit paths, locale, temporary directories, configuration, and output bounds.

Distribution track is part of capability state. The reference build uses classic F-Droid/GitHub Termux with separately installed compatible add-ons from one signing lineage. Mixing F-Droid and GitHub builds can make shared-identity add-ons incompatible. The Google Play track is a distinct product line for newer Android versions: Boot and Widget capability is merged, only part of Termux:API is built in, and Tasker is not currently the same available surface. PocketOps must report track and capability facts and refuse to claim readiness based on classic filenames alone.

Durable acceptance precedes acknowledgment. When Tasker, Widget, or a network adapter submits work, the event is validated and committed before success is returned. The producer receives an operation ID quickly. The worker later leases the event and records its attempt. A side effect uses a stable idempotency key. The result is committed before acknowledgment. If the process dies at any boundary, the ledger reveals whether to replay, reconcile, quarantine, or ask an operator.

  RECEIVED -> VALIDATED -> COMMITTED/READY -> LEASED -> EFFECT_ATTEMPTED
                                                        |
                  +-------------------------------------+----+
                  |                                          |
             RESULT_COMMITTED                           AMBIGUOUS
                  |                                          |
             ACKNOWLEDGED/DONE                   RECONCILE OR RETRY

  Retry ceiling exceeded or permanent error -> DEAD LETTER

Supervision is local recovery, not immortality. termux-services uses runit to restart a failed child while the supervisor lives. If Android kills the entire Termux process tree, runit dies too. Termux:Boot is a reboot event adapter; delivery may be delayed or blocked by Android/OEM restrictions. A short idempotent boot entry point can establish the environment, start supervision, recover expired leases, and exit. Reliability is proven by durable state and recovery evidence after destruction, not by claiming continuous uptime.

Android owns background execution. Doze and App Standby can defer network and scheduled work. Battery restrictions and OEM behavior can block or delay boot/background activity. Foreground-service rules also evolve: Android 15 introduced six-hour-per-24-hour limits for certain dataSync and mediaProcessing foreground-service types, and Android 16 applies job-runtime quotas even when jobs are started while an app has a foreground service. A Termux-hosted process cannot promise exemption. PocketOps therefore has bounded work units, explicit energy policy, offline queues, health degradation, and a product-boundary rule: if guaranteed Android scheduling/background ownership is central, move it into a conventional APK designed for current platform APIs.

Lifecycle recovery requires state-machine invariants. Graceful signal hooks are useful for ordinary stops but do not run after force-stop, kill, power loss, or some crashes. The system must remain correct before cleanup: transactions define visibility, leases expire, retries are bounded, and idempotency handles ambiguous external effects. Wakelocks may keep the CPU awake for justified visible work; they do not disable Doze or make the process immortal.

Health has multiple dimensions. Liveness asks whether an endpoint can respond. Readiness asks whether required configuration, storage, migrations, supervisor, and dependencies allow the operation. Degradation names optional capabilities that are unavailable. Recovery reports expired leases, last abnormal exit, boot evidence, backup freshness, and package/audit state. A green PID with a week-old queue is not healthy.

Security authority is decomposed. Tasker may submit only fixed typed events. SSH keys map to forced health, transfer, or emergency roles. A GUI operation has the same authorization and confirmation rules as the CLI. Network listeners default to loopback. Shared imports are copied and validated privately. Add-on calls have deadlines and permission states. Repository verification is independent of runtime authorization: a signed package can still be configured insecurely, and a secure runtime cannot compensate for unauthorized release bytes.

Mobile network truth is layered. Interface presence, DNS, TCP, TLS, and application response fail independently. Every request uses one total deadline, bounded response, explicit retry class, and battery-aware concurrency. Offline observations commit locally; synchronization is separate queued work. Replay identities prevent a reconnect from duplicating effects.

Upgrade and rollback are state transitions. P10’s package replaces owned files, verifies architecture and dependencies, and runs minimal journaled migration against private state. A bad application release may roll back only if the resulting binary understands the state schema; otherwise restoration from a verified backup is required. “Install old package” is not automatically rollback.

Backup is different from export. A user-visible report is sanitized exchange data. A disaster-recovery backup may contain irreplaceable private state and therefore needs encryption, integrity, version metadata, tested restore, and separate key recovery. The only accepted proof is a restore on a clean supported environment followed by acceptance checks.

Evidence is a product feature. Each acceptance step records plan version, environment, operation IDs, safe logs, timing, result, package/policy digest, and recovery evidence. The bundle excludes secrets, precise telemetry, private keys, full environment dumps, and unique device identifiers. Another operator can reproduce the behavior from the bundle and runbook.

Local-network policy is evolving too. If PocketOps later gains a conventional companion APK, current Android local-network permission and target-SDK requirements become that APK’s responsibility. Android 17/API 37 introduces runtime local-network permission behavior for applicable targeted apps. Do not project a companion APK’s permissions onto a Termux-hosted binary or vice versa; detect the actual component and platform contract.

2.2 Why This Matters

Integration invalidates assumptions that isolated projects can hide. A GUI may outlive a CLI invocation, Tasker may deliver duplicates, an upgrade may race a worker, an SSH health check may expose too much state, and a boot recovery may start against an incomplete migration. The capstone forces explicit ownership and ordering across these boundaries.

The result is a realistic edge-systems portfolio: a constrained node that operates offline, reports honest degradation, rejects unauthorized releases, and recovers from expected destruction. The architecture transfers to store-and-forward mobile apps, industrial gateways, field data collectors, home automation, laptop agents, and intermittently connected workers.

2.3 Historical Context / Background

Early personal automation often assumed a persistent shell process and stable filesystem. Android progressively tightened app storage, background services, broadcasts, alarms, network access, and user-visible execution in response to battery, privacy, and abuse concerns. Termux preserved a programmable Unix-style environment by operating inside an ordinary app sandbox and adding explicit companion bridges.

At the same time, reliable distributed systems converged on durable logs, idempotency, leases, retries, health, signed artifacts, and automated acceptance. PocketOps applies those production ideas to one phone without pretending the device is a datacenter server. Its small scale is useful: every trust root and state transition can be inspected.

2.4 Common Misconceptions

  • “The capstone is finished when every feature runs once.” It must survive clean install, denial, interruption, upgrade, attack, and restore.
  • “runit and Boot make an always-on server.” Both remain under Android lifecycle policy.
  • “A wakelock defeats Doze.” It is a scoped CPU-awake request, not background immunity.
  • “All Termux installation tracks expose the same bridges.” Classic and Play are distinct capability surfaces.
  • “Signed packages eliminate runtime security concerns.” Publication trust and operation authorization are separate.
  • “Rollback means reinstalling an older .deb.” State-schema compatibility or restore determines safety.
  • “A GUI should call private files directly for speed.” That creates hidden coupling and unrecoverable UI state.
  • “Connected Wi-Fi means synchronization is safe.” DNS, transport, TLS, and application layers remain unproven.
  • “A successful demo proves production suitability.” Product audience, scheduling guarantees, distribution, native permissions, and support matrix may require an APK.

3. Project Specification

3.1 What You Will Build

Build PocketOps as one signed, installable Termux product containing:

  • Doctor: environment, track, capability, permission, storage, package, service, and recovery truth.
  • Vault: private validated import/export and encrypted backup/restore boundary.
  • Telemetry broker: bounded Termux:API adapters and privacy profiles.
  • Control surfaces: Widget actions and a reconstructible Termux:GUI dashboard.
  • Network sentinel: DNS/TCP/TLS/application diagnostics and offline synchronization.
  • Restricted SSH relay: host identity, role keys, forced operations, optional controlled forwarding.
  • Tasker event bridge: fixed typed events, debounce, idempotency, and small results.
  • Supervised worker: runit, boot adapter, leases, retry, dead letter, health, and log rotation.
  • Native probe: P09 Bionic/ELF diagnostics under the shared schema.
  • Signed release: P10 package/repository, migration, staged upgrade, and tamper rejection.
  • Security gate: P11 threat model, auditor, safe evidence, and zero unsuppressed high findings.
  • Acceptance/evidence runner: a versioned plan that injects failure and creates a redacted bundle.

3.2 Functional Requirements

  1. All entry surfaces translate to versioned core operations; none implement separate business rules.
  2. Every accepted mutation commits an event/operation ID before the caller sees success.
  3. Duplicate delivery and rapid distinct events are handled by idempotency and debounce as separate mechanisms.
  4. Optional capability denial yields documented degraded behavior while required capability denial blocks only dependent operations.
  5. Android bridge calls, external commands, and network requests have deadlines and output limits.
  6. Shared storage is used only for validated import/export; code, secrets, and live state stay private.
  7. Worker, supervisor, process-tree, reboot, offline, and low-battery failures recover according to the ledger contract.
  8. SSH keys and Tasker entry points authorize only declared operations.
  9. Listeners bind to loopback unless a reviewed profile explicitly widens exposure.
  10. Package install, upgrade, migration replay, unsupported downgrade, removal, and rollback/restore behavior are defined.
  11. Repository/package tampering is rejected before execution.
  12. Security scanning is a release gate, and secret canaries never appear in evidence.
  13. Backups are encrypted, integrity-checked, versioned, and restored on a clean environment.
  14. The acceptance runner performs all 12 final-demo steps and records pass/fail with correlation IDs.
  15. The runbook states current classic/Play support and when a conventional APK is required.

3.3 Non-Functional Requirements

  • Local-first operation: core capture and queueing remain useful without network.
  • Bounded resource use: concurrency, retries, queue growth, logs, scan bytes, backup size, and benchmark duration have policies.
  • Privacy by design: precise telemetry is minimized, encrypted at rest where justified, and absent from ordinary evidence.
  • Upgrade safety: no package-owned file contains irreplaceable mutable state.
  • Auditability: every external effect links to an operation and idempotency identity.
  • Recoverability: no success depends on a graceful shutdown hook.
  • Accessibility: GUI states have text/non-color cues, safe confirmations, and reconstructible focus/view state.
  • Track honesty: unsupported bridge behavior is reported, not emulated through unsafe shortcuts.
  • Release provenance: source, dependencies, artifact hashes, signature identity, policy digest, and acceptance plan are recorded.

3.4 Example Usage / Output

$ pocketctl status
PocketOps 1.1.0  node=field-phone  schema=pocketops.status.v1
Runtime      READY     classic-fdroid aarch64 api=33
Vault        READY     objects=12 backup_age=3d
Telemetry    DEGRADED  location=permission_denied battery=ready
Network      READY     last_sync=18m queue=0
Worker       READY     runit=up restarts=1
Remote       READY     ssh=health-only listener=lan-profile
Release      READY     signed package=1.1.0
Security     READY     high=0 policy=pocketops-baseline.v1

Overall: DEGRADED (optional location capability unavailable)

$ pocketctl acceptance run --plan final-demo
01 Install from signed repository            PASS
02 Doctor on clean state                     PASS
03 Import/export boundary                    PASS
04 Telemetry partial permission              PASS
05 Widget and GUI state recovery             PASS
06 Offline queue and reconnect               PASS
07 Restricted SSH authorization              PASS
08 Tasker duplicate coalescing               PASS
09 Worker kill and lease recovery            PASS
10 Reboot and supervisor recovery            PASS
11 Upgrade, rollback, state preservation      PASS
12 Tamper rejection and security audit        PASS

Result: POCKETOPS_ACCEPTED evidence_bundle=01JZCAPSTONE

3.5 Real World Outcome

On a fresh supported Android device, an operator verifies the P10 repository key fingerprint independently, installs PocketOps, runs setup, grants only selected permissions, adds the appropriate control surface for the detected Termux track, and sees an honest dashboard. Telemetry and network observations enter one private ledger. Tasker and Widget return operation IDs for long work. A restricted SSH health key cannot open a shell. Offline events synchronize once after reconnect. Killing the worker and later the entire Termux process tree preserves work; reopening or reboot recovery safely reclaims expired leases. An upgrade preserves state; an intentionally bad application release follows the documented rollback/restore path. Tampered repository data is rejected, location revocation becomes explicit degradation, an encrypted backup restores on a clean environment, and the final P11 audit has zero unsuppressed high findings.

4. Solution Architecture

4.1 High-Level Design

                                  ANDROID OS
  +----------------------------------------------------------------------------+
  | sandbox | permissions | shared storage | Doze | app standby | OEM policy   |
  +--------------------------+----------------------+---------------------------+
                             |                      |
                  explicit bridge/entry        lifecycle events
                             |                      |
  +--------------------------v----------------------v---------------------------+
  |                         TERMUX / POCKETOPS                                  |
  |                                                                            |
  |  +-------------------- ENTRY ADAPTERS ----------------------------------+  |
  |  | shell | Widget | GUI | Tasker | SSH forced commands | Boot | webhook |  |
  |  +-------------------------------+---------------------------------------+  |
  |                                  | typed operation                         |
  |                       +----------v-----------+                             |
  |                       | contract + authz     |                             |
  |                       | validation + policy |                             |
  |                       +----------+-----------+                             |
  |                                  | durable commit                          |
  |                       +----------v-----------+                             |
  |                       | event ledger        |                             |
  |                       | queue/lease/result  |                             |
  |                       +-----+----------+----+                             |
  |                             |          |                                  |
  |                 +-----------v--+   +---v----------------+                 |
  |                 | runit worker |   | query/read models |                 |
  |                 +------+-------+   +---+----------------+                 |
  |                        |               |                                  |
  |       +----------------+---------------+-----------------------+          |
  |       |                |               |                       |          |
  |  +----v-----+    +-----v------+   +----v-----+          +------v------+   |
  |  | Android  |    | network/SSH|   | vault /  |          | native probe|   |
  |  | API      |    | sync       |   | backups  |          | Bionic ELF  |   |
  |  +----+-----+    +-----+------+   +----+-----+          +------+------+   |
  |       |                |               |                       |          |
  |       +----------------+---------------+-----------------------+          |
  |                                  |                                         |
  |                  logs + health + audit + evidence                           |
  +----------------------------------+-----------------------------------------+
                                     |
                  signed package + policy + repository trust
                                     |
                        clean install / upgrade / restore

4.2 Key Components

Component Owns Must not own
Entry adapters Transport parsing, bounded result, caller identity/context Domain policy or durable job state
Contract/auth layer Schema validation, authorization, correlation/idempotency identity Android-specific raw invocation
Event ledger Durable operations, states, leases, attempts, results Rendering/UI state
Worker/supervisor Bounded execution and local child recovery Assumption of continuous Android process life
Capability registry Track/add-on/package/permission/hardware facts Fabricated fallbacks
Vault/backup Private object/state, validation, encryption, restore metadata Executing shared-storage content
Android adapters API/notification/GUI bridge calls with deadlines Persistent business truth
Network adapters DNS/TCP/TLS/application stages and synchronization Infinite retries or implicit public binds
Release subsystem Package ownership, signing, migration, artifact provenance Runtime operator authorization
Security gate Threat model, safe findings, policy/exception state Secret collection or silent mutation
Acceptance runner Ordered assertions, failure injection, redacted evidence Special production-only behavior

4.3 Data Structures

OperationEnvelope = {
  schema, operation_id, idempotency_key,
  type, actor_role, source_surface,
  occurred_at, received_at,
  bounded_payload_ref?, requested_deadline,
  authorization_context
}

LedgerRecord = {
  operation_id,
  state: READY | LEASED | RETRY_WAIT | COMMITTED | DONE | DEAD_LETTER | AMBIGUOUS,
  lease_owner?, lease_until?, attempt_count,
  side_effect_identity?, result_ref?,
  next_attempt_at?, safe_error?, audit_refs[]
}

Capability = {
  id, track, client_state, android_component_state,
  permission_state, hardware_state, transient_state,
  status: READY | DEGRADED | BLOCKED | NOT_APPLICABLE,
  observed_at, recovery_hint
}

EvidenceBundle = {
  plan_version, environment_summary,
  package_hash, repository_key_id, policy_digest,
  step_results[], correlation_ids[],
  redaction_report, limitations,
  created_at, bundle_digest
}

Sensitive event payloads are referenced through protected private records. The general ledger and evidence bundle contain only the minimum safe attributes needed for recovery and proof.

4.4 Algorithm Overview

SUBMIT(envelope):
    authenticate/identify source surface
    validate schema, size, time, and operation authorization
    derive or verify idempotency identity
    transactionally return existing committed operation OR insert READY
    acknowledge with operation_id only after commit

WORKER_TICK(resource_budget):
    recover expired leases
    select eligible work under power/network/concurrency policy
    lease one operation transactionally
    call adapter with total deadline and side-effect identity
    classify result
    commit success before DONE, or schedule bounded retry/dead-letter/reconcile

START_OR_RECOVER(trigger):
    establish minimal explicit environment
    verify package/state schema and migration state
    start supervisor if appropriate
    recover ledger state idempotently
    report readiness/degradation and exit boot adapter quickly

ACCEPT(plan):
    verify clean baseline and trust roots
    execute each assertion/failure injection in order
    collect safe evidence by correlation ID
    abort destructive next step if recovery precondition fails
    redact, hash, and commit final bundle

Ledger operations should be indexed so a claim/lookup is approximately O(log N) rather than scanning all history. Worker concurrency is a small constant. Backup and restore are O(S) in bounded state size. Evidence generation is O(E) in captured records and must enforce a total bundle cap.

5. Implementation Guide

5.1 Development Environment Setup

Use at least two environments: a development phone and a clean supported acceptance phone or a genuinely clean Termux installation. A second computer/phone is required for SSH, listener exposure, repository bootstrap, and independent operator tests.

Before integration, record:

  • Android version/API, ABI, Termux track/version, HOME, PREFIX, and package inventory;
  • add-ons or merged Play capabilities, their source/signing compatibility, and granted permissions;
  • battery restriction state and OEM background settings;
  • repository key fingerprint verification procedure;
  • current state schema, backup key/recovery custody, and service desired state;
  • network profiles and intended bind addresses;
  • final acceptance plan version and evidence redaction policy.

Do not “normalize” the two Termux tracks by enabling unsafe external execution or copying classic add-on files. Adapt through the capability registry and state unsupported operations honestly.

5.2 Project Structure

pocketops/
|-- contracts/
|   |-- operations/
|   |-- results/
|   |-- capabilities/
|   `-- errors/
|-- core/
|   |-- ledger/
|   |-- policy/
|   |-- workers/
|   `-- read-models/
|-- adapters/
|   |-- cli-widget-gui/
|   |-- tasker-boot/
|   |-- termux-api/
|   |-- ssh-network/
|   |-- storage/
|   `-- native-probe/
|-- packaging/
|   |-- ownership/
|   |-- migrations/
|   `-- service-template/
|-- security/
|   |-- threat-model/
|   |-- policy/
|   `-- fixtures/
|-- acceptance/
|   |-- plans/
|   |-- failure-injection/
|   |-- evidence-schema/
|   `-- redaction/
|-- runbooks/
|   |-- install-upgrade.md
|   |-- incident-recovery.md
|   |-- backup-restore.md
|   `-- product-boundary.md
`-- docs/
    |-- architecture.md
    |-- support-matrix.md
    `-- privacy-model.md

5.3 The Core Question You Are Answering

Can this phone-hosted system be installed, operated, interrupted, upgraded, attacked, recovered, and explained without depending on the developer’s memory?

A complete answer is the clean-device acceptance bundle plus the runbooks and trust roots that let another operator repeat it.

5.4 Concepts You Must Understand First

  1. Stable interfaces: operation/result schemas, compatibility, streams, errors, and renderers.
  2. Android capability model: sandbox, storage, track/add-on signing, permissions, lifecycle, and product boundary.
  3. Event reliability: commit-before-ack, idempotency, debounce, leases, retry, ambiguity, and dead letter.
  4. Supervision limits: runit child recovery, Boot event delivery, process-tree death, Doze, and energy policy.
  5. Security decomposition: authentication, authorization, integrity, encryption, signing, exposure, and least privilege.
  6. Release lifecycle: package ownership, migration, signed metadata, clean install, upgrade, rollback, and restore.
  7. Acceptance evidence: failure injection, redaction, reproducibility, coverage, and residual limitations.

5.5 Questions to Guide Your Design

  1. Which component is the only owner of each state transition?
  2. What durable record exists before every external caller sees success?
  3. How is an ambiguous side effect reconciled after a crash?
  4. Can GUI, Widget, Tasker, and SSH disappear without losing the operation?
  5. Which capabilities are optional, and how does each degraded mode appear?
  6. How do track and Android-version differences change setup without changing core contracts?
  7. What happens if an upgrade begins while a worker holds a lease?
  8. Which state can roll back with package code, and which requires restore?
  9. How does the system preserve remote recovery without keeping an unrestricted key?
  10. Which evidence can be shared safely with another operator?
  11. What requirement would force a conventional APK or managed-device solution?

5.6 Thinking Exercise

Trace this event end to end:

Tasker charger event -> fixed adapter -> ledger -> worker -> telemetry adapter
-> policy decision -> notification -> result commit -> audit -> GUI status

Kill one component immediately before and after every arrow. For each kill point, write the durable state, restart trigger, duplicate risk, idempotency evidence, expected health, and operator action. Repeat once with location permission revoked and once with the phone offline.

5.7 The Interview Questions They Will Ask

  1. How did you design for Android process death?
  2. Why does runit not make a Termux service immortal?
  3. Where are PocketOps’ most important trust boundaries?
  4. How do CLI, GUI, Tasker, and SSH avoid duplicating logic?
  5. What is the difference between idempotency and debounce?
  6. How do package signing and runtime authorization differ?
  7. How do you prove a mobile network failure stage?
  8. What makes an upgrade and rollback safe?
  9. How does the clean-device acceptance test detect hidden dependencies?
  10. What would make you rebuild part or all of PocketOps as an APK?

5.8 Hints in Layers

Hint 1: Freeze public contracts

Integrate through versioned operations and results before wiring the second UI or event source.

Hint 2: One ledger

Do not let each adapter invent its own queue, retry counter, or operation identity.

Hint 3: Query, do not share internals

GUI and SSH health consume read models, not private implementation files.

Hint 4: Destroy the process early

Run kill/recovery tests before packaging so lifecycle faults do not hide inside final integration.

Hint 5: Build acceptance continuously

Add each project’s observable outcome as a versioned acceptance step rather than writing a demo script at the end.

5.9 Books That Will Help

Topic Book Suggested chapters
Schema evolution and failure “Designing Data-Intensive Applications” Ch. 4, 8, 11
Production stability “Release It!” Stability, recovery, and operations sections
Deployment/acceptance “Continuous Delivery” Acceptance and deployment pipeline chapters
Threat boundaries “Threat Modeling” Ch. 1-4
Unix process/storage/network “The Linux Programming Interface” Ch. 4-6, 20-28, 56-61

5.10 Implementation Phases

Phase 1: Architecture and support freeze

  • Declare classic reference and Play adaptation support matrix.
  • Freeze public operation/result/error schemas and state ownership.
  • Draw lifecycle, data, authority, and release boundaries.

Phase 2: Unified ledger and read models

  • Move all long work to one commit-before-ack ledger.
  • Add leases, attempts, idempotency, retry/dead-letter, and correlation.
  • Build status/read models independent of adapter internals.

Phase 3: Entry-surface integration

  • Wire CLI first, then Widget/GUI, Tasker, SSH, and Boot through fixed adapters.
  • Prove human/JSON/result limits and authorization independently.
  • Reconstruct GUI state after closure/recreation.

Phase 4: Capability and data integration

  • Connect vault, telemetry, native probe, and network sentinel.
  • Enforce private/shared storage and privacy profiles.
  • Simulate missing add-on, denied permission, unsupported hardware, offline network, and malformed response.

Phase 5: Lifecycle and energy recovery

  • Integrate runit service and short idempotent boot entry.
  • Kill worker, supervisor, and entire Termux process tree.
  • Reboot, recover expired leases, bound crash loops, and test low-battery/Doze degradation.

Phase 6: Release and migration

  • Package with P10 ownership and signing contracts.
  • Install on clean acceptance device.
  • Seed prior state, upgrade, interrupt/replay migration, and execute supported rollback/restore.

Phase 7: Security and disaster recovery

  • Run P11 against the integrated system and close high findings.
  • Create encrypted backup and restore it on a clean environment.
  • Revoke one key/permission and prove the intended degraded or denied behavior.

Phase 8: Final acceptance and handoff

  • Run all 12 steps from the signed repository and runbook.
  • Produce and independently review the redacted evidence bundle.
  • Give another operator the trust, install, recovery, and boundary runbooks.

5.11 Key Design Decisions

Decision Recommended default Reason
Core integration Versioned operation/result contracts Prevents surface-specific logic divergence
Work reliability One durable event ledger Centralizes correlation, replay, retry, and audit
Listener scope Loopback by default Network exposure must be deliberate
Android capability Runtime registry by track/component/permission/hardware Installed command alone is insufficient
Supervision runit + short Boot adapter + durable recovery Honest about Android process-tree ownership
Service energy Bounded units, backoff, low-battery degradation Avoids pretending mobile resources are unlimited
Release Signed P10 channel, service disabled by default Authentic distribution without surprise persistence
Rollback Schema-aware binary rollback or verified restore Old code may not understand new state
Security P11 zero unsuppressed high gate Makes authority reduction release-blocking
Product boundary Conventional APK when Android-native lifecycle/distribution dominates Uses the platform’s intended ownership model

6. Testing Strategy

6.1 Test Categories

  • Contract tests: all surfaces map to identical typed operations/results and stable errors.
  • Capability tests: classic/Play track, client/add-on, permission, hardware, timeout, and unsupported states.
  • Ledger tests: duplicate submit, lease expiry, crash points, retry ceiling, ambiguity, dead letter, and concurrency.
  • Storage/privacy tests: traversal, atomic import, export redaction, quota, encrypted backup, and clean restore.
  • Lifecycle tests: worker kill, supervisor kill, process-tree kill, reboot, boot restriction, Doze/offline, and low battery.
  • Network/remote tests: DNS/TCP/TLS/application failures, binds, SSH roles, forced commands, forwarding, and key revocation.
  • Release tests: clean install, hidden dependency, signed metadata, tamper, upgrade, interrupted migration, rollback, remove, and reinstall.
  • Security tests: P11 rule fixtures, canary leak tests, exception expiry, and integrated high-severity gate.
  • UX/accessibility tests: repeated taps, destructive confirmation, view recreation, non-color states, and stale status.
  • Acceptance tests: exact final plan on clean device with redacted evidence.

6.2 Critical Test Cases

Case Expected result
Location revoked while battery permission remains Partial telemetry commits; overall state degraded
Tasker repeats same delivery One logical operation/result
Three rapid distinct Wi-Fi events Debounce policy coalesces as documented
Worker dies before/after lease Unleased work remains; expired lease recovers
Worker dies after external effect, before ack Reconciliation/idempotency prevents duplicate effect
runit worker crashes repeatedly Restart is bounded/slowed and health reports crash loop
Entire Termux process tree killed Ledger remains; reopen/reboot recovery restarts safely
Boot delivery delayed/blocked Health reports supervisor down/degraded; manual recovery path works
Device offline for a day Local capture remains; bounded catch-up synchronizes once
SSH health key requests shell/forward Authorization rejects it and records safe audit
GUI closes during job Reopen reconstructs state by operation ID
Package index/artifact modified Verification rejects before execution
Upgrade interrupted Journal yields safe retry/recovery; no false new schema
Old binary incompatible with new schema Rollback refuses and uses verified restore path
Backup restored to clean device Keys/config/state follow policy and acceptance resumes
Security canary present Finding proves class without secret appearing in evidence
Classic workflow attempted on Play track Capability registry reports unsupported/adapted path

6.3 Test Data

Use versioned operation envelopes, duplicate and debounce sequences, valid/malformed Termux:API responses, permission states, layered network failures, SSH role keys with no production authority, queue crash snapshots, migration schema 1/2 fixtures, modified package/index copies, signed valid repository data, encrypted synthetic backup, canary secrets, and classic/Play capability snapshots. Every fixture has an owner, purpose, expected result, and deletion policy.

7. Common Pitfalls & Debugging

7.1 Frequent Mistakes

  1. Allowing UI or Tasker adapters to write domain tables directly.
  2. Creating separate queues and idempotency rules per surface.
  3. Depending on interactive PATH/profile in Boot, Widget, SSH, or service entry points.
  4. Treating runit uptime as Android lifecycle reliability.
  5. Running complex migration or synchronization directly in a boot script.
  6. Acknowledging work before durable result commit.
  7. Widening a listener or SSH key to make acceptance easier.
  8. Mixing classic and Play assumptions or APK signing sources.
  9. Storing backups, private keys, or executable code in shared storage unprotected.
  10. Calling an old-package install “rollback” without state compatibility proof.
  11. Producing an evidence bundle containing telemetry or secret material.
  12. Writing a final demo that bypasses actual package/authorization paths.

7.2 Debugging Strategies

  • Trace one correlation ID from entry adapter through ledger, worker, adapter, result, audit, and view.
  • Inspect durable state before inspecting terminal prose.
  • Compare environment manifests across interactive, Widget, Tasker, SSH, service, and Boot launches.
  • Diagnose network stages independently and carry one total deadline.
  • Confirm runsvdir, desired service state, child status, and logs in that order.
  • Re-run migration from a restored pre-upgrade snapshot rather than editing live state manually.
  • Verify repository signature, metadata hash, artifact hash, ELF policy, then runtime.
  • Run canary searches against the complete evidence directory after every reporting change.
  • Reduce integrated failures to contract fakes to identify adapter versus core ownership.

7.3 Performance Traps

  • Polling Android APIs or network targets while event-driven updates suffice.
  • Unlimited worker concurrency after reconnect.
  • Rapid fixed retries during outage or permission denial.
  • Rendering the entire ledger in the GUI rather than a bounded read model.
  • Running P11 full filesystem scans or P09 benchmarks as frequent health checks.
  • Re-encrypting/re-uploading unchanged backup objects.
  • Keeping unbounded logs, dead-letter payloads, or evidence bundles.
  • Holding wakelocks while queued work is waiting for network or policy.

8. Extensions & Challenges

8.1 Beginner Extensions

  • Add a read-only daily digest of readiness, queue age, package, and audit state.
  • Add a maintenance mode that stops new leases while preserving submissions.
  • Add a capability explainer that points to the exact classic/Play difference.

8.2 Intermediate Extensions

  • Create stable/candidate release channels and canary the acceptance phone.
  • Add two PocketOps profiles with different privacy and energy budgets.
  • Build a local-only Web UI over the same read models and tunnel it through the restricted SSH role.
  • Add signed configuration export/import with explicit secret exclusions.

8.3 Advanced Extensions

  • Build a two-node PocketOps Field Mesh with signed health summaries and no arbitrary remote shell control plane.
  • Add multi-ABI native probe packages and fleet capability comparison.
  • Design privacy-preserving fleet posture summaries.
  • Move lifecycle-critical scheduling into a minimal conventional companion APK while keeping a versioned Termux engine contract.
  • Model Android 17 local-network permission in that companion APK and verify the separation from Termux-hosted network behavior.
  • Add staged key rotation, repository compromise recovery, and offline-node catch-up.

9. Real-World Connections

9.1 Industry Applications

PocketOps resembles a small edge agent, offline field collector, home automation node, mobile diagnostics endpoint, or industrial gateway. Production equivalents add managed-device controls, hardware-backed keys, privacy governance, staged fleet rollout, incident response, SLOs, and a native Android component where lifecycle ownership demands it.

  • The Termux app, packages, and official add-ons provide the host ecosystem.
  • runit and termux-services supply local supervision concepts.
  • SQLite offers transaction and durability primitives suitable for the ledger.
  • OpenSSH supplies narrow remote-operation controls.
  • The Update Framework provides advanced repository compromise concepts beyond the small P10 channel.

9.3 Interview Relevance

This capstone supports systems design, mobile/Android platform, edge, SRE, security, release, and distributed-systems interviews. A strong explanation starts from an actual failure injection and traces durable state, authorization, recovery, and evidence. It also states why some production requirements would move into an APK rather than overselling Termux.

10. Resources

10.1 Essential Reading

10.2 Video Resources

  • Android Developers talks on background work, permissions, process lifecycle, and app architecture.
  • Distributed-systems talks on idempotency, queues, leases, and recovery.
  • SRE/release talks that show failure injection, staged rollout, and disaster recovery evidence.

Verify changing platform behavior against the official documentation above rather than relying on an older recording.

10.3 Tools & Documentation

  • The official Termux package-build, package-creation, and repository tools from P10.
  • OpenSSH manuals and Android application sandbox guidance.
  • SQLite transaction, WAL, backup, and integrity-check documentation.
  • Android app signing and developer-verification documentation if the product boundary moves to APK distribution.
  • Project-specific schemas, threat model, support matrix, and final acceptance plan.
  • P01 through P04 provide runtime, storage, Android bridge, and control contracts.
  • P05 through P08 provide network, SSH, Tasker, and recovery behavior.
  • P09 supplies native platform evidence.
  • P10 supplies signed installation and upgrade.
  • P11 supplies the security release gate.

11. Self-Assessment Checklist

11.1 Understanding

  • I can explain Termux’s Android/Bionic/PREFIX model and installation-track differences.
  • I can trace an event through commit, lease, effect, result, acknowledgment, and recovery.
  • I can explain what runit, Boot, wakelocks, and Doze do and do not guarantee.
  • I can distinguish runtime authorization, transport security, package signing, and APK signing lineage.
  • I can explain why rollback may require state restore.
  • I can identify the point where an APK becomes the responsible product boundary.

11.2 Implementation

  • All surfaces use versioned core operations and one ledger.
  • Capability denial produces correct degraded or blocked states.
  • Offline, duplicate, kill, reboot, and retry cases recover without duplicate logical effects.
  • Clean install and upgrade use only signed repository artifacts and declared dependencies.
  • Backup restores on a clean supported environment.
  • SSH, Tasker, listeners, secrets, packages, and logs pass P11’s high-severity gate.
  • The final evidence bundle is reproducible and contains no canary secret.

11.3 Growth

  • I can propose a two-node rollout with canary promotion and key revocation.
  • I can move one lifecycle-critical function into an APK without duplicating the core contract.
  • I can state PocketOps’ residual risks, unsupported devices/tracks, and operational limits.
  • I can hand installation, incident recovery, and restore to another operator.

12. Submission / Completion Criteria

Minimum completion

  • Doctor, vault, one Android bridge, event ledger, supervised worker, and signed package integrate through stable contracts.
  • Offline submission and one worker-kill recovery complete without duplicate logical effects.
  • A clean supported environment installs from signed metadata.
  • The threat model and acceptance plan are written before the final demo.

Full completion

  • All twelve components and functional requirements are integrated.
  • The exact 12-step acceptance plan passes on a clean supported device.
  • Permission denial, missing capability, offline operation, worker/supervisor/process-tree death, reboot, duplicate delivery, and low-battery degradation are evidenced.
  • Upgrade, interrupted/repeated migration, supported rollback or restore, encrypted backup restore, removal, and reinstall follow the runbook.
  • Tampered index/package, unauthorized SSH/Tasker requests, and unsafe listeners are rejected.
  • P11 reports zero unsuppressed high findings and canary secrets are absent from the complete evidence bundle.
  • The support matrix distinguishes classic and Play tracks and documents Android lifecycle limits.

Excellence criteria

  • A second operator completes install, failure injection, recovery, and handoff using only trust roots and runbooks.
  • Stable/candidate channels demonstrate canary promotion and rollback.
  • A second clean device or installation validates backup portability and hidden-dependency absence.
  • The learner produces an APK-boundary design for one lifecycle-critical feature, including local-network permission implications, without implementing duplicate domain logic.
  • A two-node Field Mesh demonstrates offline catch-up, key revocation, and signed health exchange without granting a central arbitrary shell.

Generated from the Termux Android Apps and Tools Mastery parent guide and its expanded project index. This guide intentionally provides architecture, pseudocode, metadata, tests, and observable contracts rather than complete runnable source code.