Project 4: Durable Automation Library
Persist rules and execution evidence so user intent survives process death, schema evolution, import failure, and uncertain side effects.
Quick Reference
| Attribute | Value |
|---|---|
| Difficulty | Level 2 — Intermediate |
| Time Estimate | 18-26 hours |
| Language | Kotlin; Java and SQL are useful alternatives for persistence exercises |
| Prerequisites | Projects 1-3, relational data basics, transactions, serialization concepts |
| Key Topics | Room, SQLite, DataStore, migrations, execution journal, import/export, recovery, retention |
1. Learning Objectives
By completing this project, you will:
- Separate durable desired automation state from observed execution history.
- Design Room entities without persisting Android objects or implementation class names.
- Distinguish database schema version, rule-language version, and export-package version.
- Use transactions to claim action attempts before effectful work begins.
- Classify restart recovery as safe retry, idempotency query, permanent failure, or ambiguous.
- Test migrations with real previous-version database snapshots.
- Validate and stage imports before one atomic merge or replacement.
- Exclude secrets, ephemeral tokens, and prohibited content from backup and diagnostics.
- Apply retention, compaction, deletion, and corruption behavior deliberately.
2. Theoretical Foundation
2.1 Core Concepts
Desired state versus observed state
Rules, schedules, selected capability scopes, and plugin references express what the user wants. Runs, plans, attempts, results, and health transitions describe what happened. Mixing them creates destructive bugs: deleting run history must not delete a rule, and editing a rule must not rewrite historical evidence.
Persistence boundary
Room maps stable storage records to domain data. The repository owns transactions and translation. The rule engine does not know SQL, Cursor, Entity, or Room annotations. Android adapters do not store arbitrary framework objects. Stable type IDs and versioned configuration preserve meaning through refactors.
Three kinds of versioning
Database version controls tables, columns, indexes, and constraints. Rule-language version controls trigger, condition, action, and policy semantics. Export version controls the interchange package, manifest, checksums, and merge behavior. A database migration can preserve a rule whose language node still requires a separate semantic migration.
Execution journal
Before a side effect, the runtime records an immutable plan and transactionally claims an attempt. After the adapter returns, it finalizes success, retryable failure, permanent failure, user action, or ambiguity. A process can die between every transition, so restart logic derives recovery from journal state rather than memory.
Transactional import
An import is untrusted data. Parse into staging, bound its size, validate structure, verify checksum if present, migrate package and rule nodes, resolve unknown references, compute a preview, and only then commit one transaction. One invalid rule must not partially overwrite a valid library.
Data minimization and retention
Execution traces can become behavioral surveillance. Persist typed evidence instead of raw platform objects. Secret material belongs behind Keystore-backed handles or user re-entry, not in export files. Retention distinguishes recent detailed traces, older summaries, aggregate metrics, and user deletion.
2.2 Why This Matters
Android can kill an automation process without an orderly shutdown. Reboots discard in-memory registrations. Plugins can disappear. Users edit rules while scheduled work waits. A system built around mutable singleton state eventually loses intent or repeats effects.
Durability does not mean persisting everything. The library preserves facts needed for recovery and explanation while minimizing sensitive content. It also makes future projects possible: WorkManager receives stable IDs, plugins can become unavailable without destroying their nodes, and support tools can replay a redacted event against the current planner.
Migration discipline protects user trust. A rule library may outlive many application releases. Automatic destructive migration is unacceptable because automations encode time, behavior, and sometimes safety expectations.
2.3 Historical Context / Background
Relational databases provide transactions, constraints, indexes, and mature recovery semantics. Room adds compile-time query verification and Android integration while retaining SQLite behavior. DataStore suits small preference-like state but does not replace relational transactions for rules and attempts.
Enterprise persistence patterns such as Repository, Data Mapper, Unit of Work, and optimistic concurrency remain relevant. Event sourcing concepts help distinguish immutable facts from current projections, even if this project does not implement a full event-sourced database.
2.4 Common Misconceptions
- Room migration equals rule migration: storage layout and language semantics evolve independently.
- JSON is automatically forward compatible: unknown types and changed semantics still need policy.
- Persisting a Kotlin class is convenient: class/package renames are not stable contracts.
- A transaction can make HTTP atomic: local commit and external effect remain separate systems.
- Success not recorded means failure: process death after an accepted effect creates ambiguity.
- Encrypted storage makes export safe: exported secrets may bypass the original storage protection.
- DataStore is a general database: it lacks the relational journal and transaction model needed here.
3. Project Specification
3.1 What You Will Build
Build a Room-backed automation library with:
- Rule catalog and ordered rule graph storage.
- Capability dependency and unavailable-node records.
- Immutable plan, run, action-attempt, and result journal.
- Migration suite using real database snapshots.
- Versioned import/export package with preview and redaction.
- Retention and user deletion controls.
- Process-death recovery classifier.
- Compose Library and Run History screens.
Do not execute real external side effects in this project. Simulated adapters and journal transitions are sufficient.
3.2 Functional Requirements
- Rule lifecycle: Create, duplicate, reorder, enable, disable, soft-delete or delete, and restore.
- Stable schema: Persist type IDs, versions, bounded configuration, and relationships.
- Historical snapshot: Runs reference the rule revision and plan that actually executed.
- Attempt claim: Only one active claim exists for a plan action and idempotency identity.
- Recovery classifier: Identify pending, running, retryable, permanent, succeeded, ambiguous, or user-action states.
- Migration: Upgrade at least one real previous schema fixture without destructive fallback.
- Import staging: Reject malformed, oversized, incompatible, or partially invalid packages atomically.
- Export preview: List included categories and redacted fields before sharing.
- Secret exclusion: Never export Keystore material, ephemeral PendingIntent-like tokens, credentials, or raw sensitive text.
- Retention: Purge detailed traces according to user-visible policy while preserving necessary summaries.
- Unknown nodes: Preserve configuration in a disabled state when implementation is absent.
- Process recreation: Reconstruct UI and recovery state from repositories.
3.3 Non-Functional Requirements
- Integrity: Foreign keys, uniqueness, transactions, and checks validate state transitions.
- Privacy: Redaction occurs before persistence when raw data is unnecessary.
- Reliability: No destructive migration fallback is used for user automation data.
- Performance: Library lists and history use indexed, paged, bounded queries.
- Usability: Migration and import failure explain what remains unchanged.
- Testability: Repositories accept injectable clocks and simulated crash boundaries.
3.4 Example Usage / Output
IMPORT PREVIEW
Package version: 2
Rules found: 4
Rules valid: 3
Rules unavailable: 1 plugin action missing
Secrets included: 0
Ephemeral tokens included: 0
Conflicts: 1 existing rule name
Decision: NOT COMMITTED
Reason: rule 4 contains unsupported schema version 9
Existing library changes: 0
3.5 Real World Outcome
Open the Library screen. Create the low-battery demo rule, duplicate it, reorder both entries, and disable one. Force-stop the app and reopen. The same rule order, enabled state, versions, and capability dependencies appear.
Run a simulated plan. The Run History screen shows correlation identity, rule revision, event summary, context snapshot identity, planned action, claim, and final simulated result. Trigger a debug crash after claim but before finalization. Reopen the app; the recovery screen identifies the attempt as interrupted and explains whether it can retry or must become ambiguous.
Install or open a version-one database fixture. Room applies the migration and a report states which schema and rule-language transformations occurred. The logical rule remains equivalent. No destructive fallback erases the database.
Export the library. A preview lists rules, metadata, history range, and excluded secret categories. Inspect the package and verify no seeded secret marker appears. Corrupt one node in a multi-rule package and import it. Validation rejects the package before commit; all existing rules remain unchanged.
Remove support for a fake plugin action. Its stored node remains visible as unavailable, preserving configuration for future reinstall instead of crashing or disappearing.
4. Solution Architecture
4.1 High-Level Design
RuleForge domain Compose screens
| |
| ports v
+--------------------> Repository interfaces
|
v
+------------------+
| Room transaction |
| boundary |
+---+----+----+----+
| | |
rules plans attempts/results
| | |
+----+----+
|
+------------+-------------+
| |
recovery classifier retention worker
|
v
safe retry / query /
ambiguous / complete
import bytes -> staging -> validate -> migrate -> preview -> one commit
export request -> projection -> redact -> manifest/checksum -> preview
4.2 Key Components
| Component | Responsibility | Key Decisions |
|---|---|---|
| RuleRepository | Transactional rule catalog and revisions | Domain models across boundary |
| RunJournalRepository | Plans, claims, attempts, and outcomes | Append facts; bounded corrections |
| SchemaMigrator | Database layout evolution | Real snapshot tests |
| RuleLanguageMigrator | Domain semantic evolution | Explicit node transformations |
| ImportStager | Parses and validates untrusted package | No live mutation |
| ExportProjector | Selects and redacts exportable data | Allowlist fields only |
| RecoveryClassifier | Maps interrupted journal to next safe state | Never assumes external failure |
| RetentionManager | Compacts and deletes history | User-visible policy |
| SecretReferenceStore | Persists aliases/metadata, not raw secret export | Keystore boundary |
4.3 Data Structures
RuleRecord
rule_id
current_revision
enabled
position
created_at
updated_at
RuleRevisionRecord
rule_id
revision
language_version
canonical_graph
graph_hash
PlanRecord
plan_id
correlation_id
event_summary
context_snapshot_id
rule_revision
status
AttemptRecord
attempt_id
plan_id
action_index
idempotency_key
state
claimed_at
finished_at optional
outcome_reason optional
4.4 Algorithm Overview
Key Algorithm: Recover Interrupted Attempt
- Load plan, action descriptor, last attempt, and durable state.
- If no claim exists, the action has not begun and can be claimed normally.
- If a terminal result exists, do nothing.
- If claimed but no external invocation evidence exists, apply the adapter’s safe retry policy.
- If invocation may have crossed the boundary, check idempotency-query capability.
- If the external target confirms the idempotency key, finalize from evidence.
- If target rejects or proves no effect, schedule a bounded retry when permitted.
- If outcome cannot be proven and repetition may be unsafe, mark Ambiguous and require review.
- Persist recovery decision transactionally before any next attempt.
Complexity Analysis:
- Time: O(1) per attempt recovery with indexed plan/action identities; O(R) for scanning R incomplete attempts at startup.
- Space: O(R) for the recovery working set, bounded by paging.
5. Implementation Guide
5.1 Development Environment Setup
Use Room, SQLite inspection tools, DataStore only for small preferences, Android Keystore for later secret-reference exercises, and the Room migration test helper. Prepare a version-one database fixture committed to test resources.
$ ./gradlew :automation-storage:testDebugUnitTest :automation-storage:connectedDebugAndroidTest
Migration 1 -> 2: PASSED
Import atomicity: PASSED
Seeded secret scan: 0 matches
This is a target verification transcript, not a runnable solution.
5.2 Project Structure
durable-automation-library/
├── storage-domain/
│ ├── repositories/
│ ├── recovery/
│ └── retention/
├── storage-room/
│ ├── entities/
│ ├── dao/
│ ├── mappings/
│ ├── migrations/
│ └── database/
├── interchange/
│ ├── staging/
│ ├── validation/
│ ├── migration/
│ └── export/
├── app/ui/library/
├── app/ui/history/
└── tests/fixtures/databases/
5.3 The Core Question You Are Answering
Which facts must survive process death, and how do I evolve them without changing what the user’s automation means?
Answer separately for user intent, historical evidence, secrets, ephemeral authority, and derived caches. Not every value deserves durability.
5.4 Concepts You Must Understand First
Stop and research these before coding:
- Transactions and invariants
- Which changes must commit together?
- What makes an attempt claim unique?
- Book Reference: Patterns of Enterprise Application Architecture — Unit of Work.
- Repository and Data Mapper
- Why should Room entities stay outside RuleForge?
- Book Reference: Patterns of Enterprise Application Architecture — Repository and Data Mapper.
- Schema versus semantic migration
- Can a column migrate while a node remains semantically obsolete?
- Book Reference: Refactoring Databases — evolutionary migration.
- External-effect ambiguity
- Which crash window makes retry unsafe?
- What evidence can an idempotent receiver provide?
- Data classification
- Which fields are public, sensitive, secret, or prohibited?
- Which fields can be hashed or summarized before persistence?
- Backup boundaries
- What does Android backup include by default?
- Which database or files require exclusion rules?
5.5 Questions to Guide Your Design
- Which entities represent desired state and which represent history?
- Does every run reference an immutable rule revision?
- What constraint prevents duplicate action claims?
- How is unknown plugin configuration retained?
- How does an import conflict resolve without silent overwrite?
- Which traces age into summaries?
- How does user deletion interact with required operational records?
- Can a failed migration leave the old database intact?
- Which values belong in DataStore rather than Room?
- How does export prove that secrets are absent?
5.6 Thinking Exercise
Trace a Crash Around an Accepted Webhook
Draw this sequence:
plan committed
-> attempt claimed
-> request sent with idempotency key
-> receiver accepts effect
-> process dies
-> no local success result
-> app restarts
Questions to answer:
- Which rows exist at restart?
- May the app resend immediately?
- Can it query the receiver by idempotency key?
- What if the receiver has no query API?
- Which message should the user see?
- Which state is fact versus inference?
5.7 The Interview Questions They Will Ask
- Why use a transaction around an execution claim?
- How do Room migrations differ from JSON import migrations?
- What is forward compatibility for an unknown rule node?
- How do you keep secrets out of backups and exports?
- What happens if the process dies after an effect but before commit?
- How would you recover from a failed migration?
5.8 Hints in Layers
Hint 1: Separate desired and observed state
Rule rows are not run-history rows.
Hint 2: Persist type IDs, not code identity
A refactor should not rewrite user meaning.
Hint 3: Stage imports completely
Parsing and preview occur before mutation.
Hint 4: Kill at every journal edge
Recovery semantics emerge only when interruption is deliberate.
5.9 Books That Will Help
| Topic | Book | Chapter |
|---|---|---|
| Persistence patterns | Patterns of Enterprise Application Architecture | Repository, Unit of Work, Data Mapper |
| Migration | Refactoring Databases | Evolutionary database design |
| Domain boundaries | Clean Architecture | Boundaries |
| Messaging recovery | Enterprise Integration Patterns | Idempotent Receiver |
| Privacy | Foundations of Information Security | Data protection |
5.10 Implementation Phases
Phase 1: Rule Catalog and Mappings (5-7 hours)
Goals:
- Persist stable rule revisions and catalog state.
- Keep Room entities out of the domain.
Tasks:
- Define entities, constraints, indexes, and mapping ports.
- Implement create, duplicate, reorder, enable, and disable transactions.
- Build the Compose Library screen and process-recreation tests.
Checkpoint: Force-stop and reopen preserves one logical rule and order.
Phase 2: Journal and Recovery (6-9 hours)
Goals:
- Store plans and claims safely.
- Classify interrupted outcomes.
Tasks:
- Define immutable plan and attempt records.
- Implement unique claim and terminal transition invariants.
- Add debug crash points and recovery classifier fixtures.
Checkpoint: Every interruption window maps to safe retry, terminal, or Ambiguous.
Phase 3: Migrations, Import/Export, and Retention (7-10 hours)
Goals:
- Prove evolution and atomic interchange.
- Bound sensitive historical data.
Tasks:
- Add a real version-one database and migration report.
- Build staging, validation, preview, and one-transaction import.
- Add export allowlist, seeded-secret scan, retention, and deletion.
Checkpoint: A corrupted package changes zero live rows and a real snapshot upgrades logically.
5.11 Key Implementation Decisions
| Decision | Options | Recommendation | Rationale |
|---|---|---|---|
| Rule representation | Class blob; stable graph data | Stable graph data | Evolvable and inspectable |
| History | Mutate latest; immutable transitions | Immutable facts plus status projection | Recovery evidence |
| Import | Update while parsing; stage then commit | Stage then commit | Atomicity |
| Migration failure | Destructive fallback; fail safely | Fail safely | Preserve user intent |
| Secrets | Raw columns; encrypted export; external handle | Keystore-backed handle | Avoid export leakage |
| Retention | Infinite; delete all; tiered | Tiered user-visible policy | Balance support and privacy |
6. Testing Strategy
6.1 Test Categories
| Category | Purpose | Examples |
|---|---|---|
| DAO tests | Prove constraints and queries | Stable order and unique claims |
| Repository tests | Prove transaction boundaries | Duplicate and enable atomically |
| Migration tests | Prove real prior data survives | Version-one snapshot to current |
| Recovery tests | Prove crash semantics | Death after claim or effect evidence |
| Interchange tests | Prove staging and redaction | Corrupt node and seeded secret |
| UI tests | Prove understandable states | Unavailable node and import preview |
| Retention tests | Prove bounded history | Detail compacted, summary retained |
6.2 Critical Test Cases
- Process death: Rules and revisions reconstruct with logical equality.
- Unique claim: Concurrent claim attempts produce one active winner.
- Terminal immutability: Succeeded attempt cannot silently return to pending.
- Ambiguous crash: External invocation evidence without result never auto-retries unsafe action.
- Real migration: Prior snapshot retains rule meaning and history linkage.
- Failed migration: Original database remains recoverable; no destructive fallback.
- Atomic import: One invalid node causes zero live-library changes.
- Unknown plugin: Configuration remains visible and disabled.
- Secret scan: Unique marker occurs zero times in export and support projection.
- Retention: User-selected purge removes eligible detail and updates UI truthfully.
- Large package: Size and depth bounds reject resource exhaustion before commit.
6.3 Test Data
rule fixture v1
rule_id: evening-reminder
language_version: 1
trigger: power.connected v1
action: notification.demo v1
migration expectation
database_version: 1 -> 2
rule_language: 1 -> 2
logical_trigger: unchanged
logical_action: unchanged
audit_report: one explicit transformation
import corruption
rules: 3 valid, 1 unsupported node version
expected_live_mutations: 0
7. Common Pitfalls & Debugging
7.1 Frequent Mistakes
| Pitfall | Symptom | Solution |
|---|---|---|
| Class-name storage | Rename breaks saved rules | Stable type IDs and migrators |
| Partial import | Some rules change before error | Stage and one transaction |
| Destructive fallback | Upgrade erases library | Explicit migrations and fail safely |
| Rule/history mixing | Deleting logs removes rules | Separate desired and observed tables |
| Retry assumption | Duplicate external effect | Journal idempotency and ambiguity |
| Export blacklist | New secret field leaks | Export allowlist and schema classification |
| Huge trace queries | Library screen stalls | Index, page, summarize, retain |
7.2 Debugging Strategies
- Inspect database invariants: Use Database Inspector after controlled transitions.
- Compare canonical graphs: Verify logical meaning rather than row layout.
- Stop at journal edges: Inject process termination before and after each transition.
- Use real snapshots: Synthetic schema creation can miss migration history defects.
- Preview staging: Display validation errors before a transaction begins.
- Seed secrets: Search database projections, exports, logs, and support bundles for a unique marker.
7.3 Performance Traps
Do not load complete rule graphs and full traces for every list row. Store summary projections and indexes for enabled state, position, last result, and next run. Page history. Bound graph depth and import size. Batch retention deletions so they do not monopolize database access. Measure migration time on a realistic fixture.
8. Extensions & Challenges
8.1 Beginner Extensions
- Add rule tags and filtered catalog views.
- Add export of one selected rule without history.
- Add a migration-report detail screen.
8.2 Intermediate Extensions
- Add optimistic revision checks to prevent lost edits.
- Add merge preview with rename, replace, and skip decisions.
- Add compact history summaries after detailed retention expires.
8.3 Advanced Extensions
- Add encrypted local fields represented by non-exportable secret aliases.
- Add tamper-evident export manifests with checksums and bounded signatures.
- Build a recovery model checker over all journal state transitions.
- Benchmark migration and paging with a large synthetic library.
9. Real-World Connections
9.1 Industry Applications
- Workflow engines: Persist definitions separately from runs and attempts.
- Payment orchestration: Journal ambiguous external effects and idempotency.
- Sync clients: Reconcile desired state after process death and reboot.
- Password managers: Keep secret material outside ordinary exports.
- Enterprise agents: Migrate long-lived policy and audit data safely.
9.2 Related Open Source Projects
- Android Architecture Samples: https://github.com/android/architecture-samples — repository and persistence patterns.
- Now in Android: https://github.com/android/nowinandroid — Room, DataStore, and layered Android architecture.
- Room source and samples: https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:room/ — implementation and test references.
9.3 Interview Relevance
- Transactions and uniqueness show practical database integrity knowledge.
- Version separation demonstrates mature schema evolution.
- Ambiguous recovery connects local persistence to distributed effects.
- Import staging and secret exclusion reveal security-aware product design.
10. Resources
10.1 Essential Reading
- Save data with Room: https://developer.android.com/training/data-storage/room — database architecture and entities.
- Room migrations: https://developer.android.com/training/data-storage/room/migrating-db-versions — automatic/manual migration and testing.
- DataStore: https://developer.android.com/topic/libraries/architecture/datastore — small durable preference state.
- Data backup: https://developer.android.com/identity/data/autobackup — backup rules and exclusions.
- Android Keystore: https://developer.android.com/privacy-and-security/keystore — non-exportable key operations.
- App architecture data layer: https://developer.android.com/topic/architecture/data-layer — repository boundaries.
10.2 Video Resources
- Android data and storage sessions: https://www.youtube.com/@AndroidDevelopers
- Room learning resources: https://developer.android.com/training/data-storage/room
10.3 Tools & Documentation
- Database Inspector: https://developer.android.com/studio/inspect/database — inspect live Room state.
- Room migration testing: https://developer.android.com/training/data-storage/room/migrating-db-versions#test — real fixture validation.
- App Inspection: https://developer.android.com/studio/debug/app-inspection — runtime database tooling.
- Android security best practices: https://developer.android.com/privacy-and-security/security-best-practices — data handling controls.
10.4 Related Projects in This Series
- Previous — Project 3: RuleForge Core: Defines the stable domain language persisted here.
- Next — Project 5: Timekeeper Scheduler: Stores desired schedules and journaled delivery.
- Project 10: Plugin SDK: Exercises unavailable nodes and versioned external types.
- Project 14: Automation Observatory: Reads redacted journal evidence and retention projections.
11. Self-Assessment Checklist
Before considering this project complete, verify:
11.1 Understanding
- I can distinguish desired state from execution history.
- I can distinguish database, rule-language, and export versions.
- I understand why a local transaction cannot make an external effect atomic.
- I can classify interrupted attempts as retryable, terminal, or ambiguous.
- I know which data must never enter an export.
- I can explain why real snapshot migration tests matter.
11.2 Implementation
- Rule catalog and immutable revisions survive process death.
- Plans, claims, attempts, and results have explicit schemas.
- Unique claim and terminal-state invariants are enforced.
- A real prior database migrates without destructive fallback.
- Import is staged, validated, previewed, and atomic.
- Unknown nodes remain visible and disabled.
- Seeded secrets never appear in export or support projections.
- Retention and deletion behavior is user-controlled and tested.
11.3 Growth
- I documented one recovery state that I initially misclassified.
- I can whiteboard the execution journal in an interview.
- I measured one realistic migration or history query.
- I wrote a retrospective about durability versus data minimization.
12. Submission / Completion Criteria
Minimum Viable Completion:
- Create, duplicate, reorder, enable, and disable rules in Room.
- Rules and one simulated run survive force-stop and reopen.
- Plan and attempt claims use explicit transaction boundaries.
- One real database migration fixture passes.
- One invalid multi-rule import changes zero live rows.
Full Completion:
- All minimum criteria plus:
- Recovery classifies every documented process-death boundary.
- Export preview and seeded-secret scan pass.
- Unknown plugin node configuration remains available but disabled.
- Retention, compaction, and deletion tests pass.
- Database, rule-language, and export versions are independently documented.
Excellence — Going Above & Beyond:
- Build a model-based test over the complete attempt state machine.
- Produce migration performance evidence on a large fixture.
- Add tamper-evident, bounded export metadata without exporting secrets.
- Demonstrate an idempotency-query recovery versus an explicitly Ambiguous recovery.
This guide was generated from ANDROID_AUTOMATION_APPS_MASTERY.md. For the complete learning path, see the parent guide and directory README.