Durability
What persists in Yoke, when writes land, crash-safe dequeue, and storage backend guarantees.
Yoke provides durability by default — no opt-in checkpoints, no explicit flush calls. Understanding what persists and when helps you reason about failure modes and design reliable systems.
Three durable layers per actor
| Layer | Storage | Survives restart |
|---|---|---|
| Queue | Postgres actor_queues | Yes |
| SQLite state | Postgres actor_sqlite (BYTEA) or S3 | Yes |
| Instance metadata | Postgres actor_instances | Yes |
Plus the global stream in stream_events — append-only audit log for all events.
When writes persist
| Operation | Persisted when |
|---|---|
ctx.append() | Immediately — row in stream_events before return |
ctx.state.get() | Read — no write |
ctx.state.set() | Immediately — SQLite write + backend flush |
ctx.state.update() | Immediately — transaction + state_cas optimistic lock |
ctx.sqlite.exec() | Immediately — direct DB write |
ctx.conversations.postMessage() | Atomically — Postgres INSERT + stream append |
| Queue dequeue | After process() returns successfully |
| Schedule upsert | Immediately — Postgres schedules row |
ctx.state is not an in-memory object. It is an async handle backed by SQLite — every get(), set(), and update() round-trips to the storage backend.
Crash-safe processing
The dequeue semantics are the durability contract for event handling:
1. Event enqueued to actor_queues (status: pending)
2. Scheduler claims event (status: processing)
3. Deno runs process(ctx, event)
4. SUCCESS → row deleted from queue
5. THROW → status reverted to pending, retry laterThere is no early-ack. If process() runs for five minutes, the event occupies a concurrency slot for five minutes. Structure long work as:
- Multiple events (chained via correlation)
- Schedules for delayed continuation
- Child instances for parallel work
Atomic state updates
ctx.state.update() uses optimistic concurrency:
await ctx.state.update((s) => ({
...s,
jobsCompleted: s.jobsCompleted + 1,
}));Under the hood: read version → apply updater in Deno → state_cas op with expected version → SQLite transaction. Concurrent updaters serialize; no lost writes.
Direct ctx.state.set() overwrites without version check — use update() for read-modify-write.
Storage backends
Per-instance SQLite persistence is controlled by YOKE_STORAGE:
| Value | Behavior |
|---|---|
pg_sqlite | Serialized SQLite databases stored as BYTEA in Postgres actor_sqlite table |
s3 | SQLite files in S3 (YOKE_S3_BUCKET); local cache dir for rusqlite execution only |
pg_sqlite stores BYTEA blobs in Postgres — it is not the PostgreSQL pg_sqlite extension. SQLite runs locally in the worker process; the serialized file is persisted to Postgres.
The SDK surface (ctx.state, ctx.sqlite) is identical in both modes.
S3 sync modes
YOKE_S3_SYNC | Behavior |
|---|---|
lazy (default) | Upload after writes; flush_all() on shutdown |
strict | fsync local state.db before each S3 upload |
With strict, a successful state_set / sqlite_exec / state_cas implies the local file is durable before upload begins. A crash between local write and upload may lose data in lazy mode.
Stream durability
Stream appends are transactional in Postgres. Once ctx.append() resolves, the event has a seq and will fan-out (possibly after a short async delay).
Fan-out delivery is idempotent — fanout_delivered prevents duplicate enqueues for the same (tenant_id, stream_seq, instance_id).
Conversation durability
Messages are canonical in Postgres. conversation.message.posted stream events carry ids only — handlers read full content from the conversation store when needed.
Kernel ops perform INSERT + stream_append atomically — user code cannot post to a conversation without persisting it.
What is not durable
| Ephemeral | Why |
|---|---|
In-flight process() local variables | Lost on crash — recover from queue |
Deno module scope without ctx.state | Worker restart clears memory |
Un-awaited promises in process() | May not complete before crash |
Always persist important data via ctx.state, ctx.sqlite, or ctx.append() before returning from process().
Failure recovery
On restart:
- Load instance records from Postgres
- Rebuild fan-out index from persisted filters +
filterFromConfig - Provision SQLite from
actor_sqliteor S3 - Resume drain loops — pending queue events reprocessed
- Portal
start()loops restart
No stream replay. Actors pick up where their queues left off.