Skip to content

Durability and disconnected devices

The current architecture is a single durable broker with leased workers, not a Kafka or Temporal installation. Those systems would not eliminate the need for device-local persistence and idempotent result acceptance.

What survives

  • Broker restart: jobs, accepted results, enrollment hashes and audit events survive in SQLite. Preserve OPEN_COMPUTE_DB and both broker secrets.
  • Desktop worker restart after computation: a mode-0600 *.outbox.json beside its credential file stores the completed result before transmission. The worker retries this result before claiming any new task.
  • Browser reload after computation: the result is stored in same-origin local storage. It is retried after the user consents and starts again. Clearing site storage or private-session teardown removes this protection.
  • Lost acknowledgment: identical output from the same worker and lease is acknowledged again without another completion event or proof credit.
  • Long outage: the old lease expires. A subsequent claim requeues the task; after three total claims it becomes failed. A stale outbox is rejected and cleared. This is at-least-once execution, not exactly-once computation.

Python workers back off with jitter on failures and tolerate transient heartbeat failures within their last confirmed lease. Browser retries run only while the user remains opted in and the page is visible. Stop prevents subsequent claims and retries; it cannot undo an HTTP result already accepted by the broker.

Limits and operations

An unfinished computation is not checkpointed by this alpha. On process/device failure it is repeated elsewhere after expiry. Real training profiles must use their own durable checkpoint storage, bounded Slurm time limits and idempotent output paths. A worker crash can leave a Slurm job alive and cause duplicate work.

Use one running process per credential/outbox file. Treat the outbox as sensitive: it contains a lease credential and result, in plaintext. On Windows protect the directory with NTFS ACLs; POSIX mode bits alone do not establish Windows access control. SQLite is not replicated or an HA service. Back up the database while stopped; local disk loss still loses state. File replacement and SQLite commits are not a substitute for tested storage-level power-loss recovery.

When to add another system

For multi-broker operation, first move transactional state to a shared database such as PostgreSQL. Temporal is a possible later coordinator for multi-step workflows, timers and compensation. Kafka is appropriate for high-volume event streams and downstream consumers, not as a replacement for lease ownership or job-state transactions. Neither is required to run this alpha.