Rust Runtime
Overview of the Yoke Rust kernel — Postgres-backed stream, fan-out, queues, scheduler, storage, and embedded Deno.
The Rust runtime is Yoke's kernel. It owns durable storage, the global event stream, fan-out to actor queues, actor scheduling, portal HTTP, and the embedded Deno worker that runs user TypeScript.
Application builders interact with Yoke through the TypeScript SDK. The runtime is what makes append-only events, filter-based delivery, and per-instance SQLite actually durable across restarts.
Responsibilities
| Subsystem | Role |
|---|---|
| Orchestration | YokeRuntime::start, op handler, boot |
| Stream | Per-tenant sequenced append to stream_events |
| Fan-out | Two-tier filters → actor_queues |
| Queues | Persisted per-instance work queues |
| Scheduler | Drain loops, concurrency, Deno dispatch_process |
| Storage | Per-instance SQLite (pg_sqlite or S3) |
| Deno | In-process TypeScript via libdeno + JSON-RPC |
| HTTP | Portal routes, /health, /metrics |
Stack position
User package (yoke.ts)
↓ load / dispatch
Embedded Deno worker
↕ newline-delimited JSON-RPC (unix socket)
YokeRuntime (Rust)
→ Postgres — stream, instances, queues, conversations
→ SQLite — per-actor state (pg_sqlite BYTEA or S3 objects)
→ HTTP — portal ingress/egressCore invariants
- Stream is the spine — all communication is append-only events; no direct actor channels.
- Append never blocks on fan-out — fan-out runs asynchronously after the row is committed.
- No stream read API for user code — only the kernel tails the stream internally.
- Crash-safe dequeue — events leave the queue only after
process()returns successfully in Deno. - Tenant isolation —
tenant_idscopes stream seq, instances, queues, and fan-out idempotency.
Postgres tables (runtime)
| Table | Purpose |
|---|---|
stream_events | Global append-only event log (tenant_id, seq) |
tenant_stream_counters | Monotonic seq allocator per tenant |
actor_instances | Actor instance registry (config, filter, queue policy) |
portal_instances | One portal instance per definition per tenant |
actor_queues | Per-instance pending/processing events |
dead_letters | Saturated or exhausted-retry events |
actor_sqlite | Serialized SQLite files (store_id, instance_id, db) |
fanout_delivered | Idempotency ledger for fan-out per (tenant_id, stream_seq) |
runtime_workers | Horizontal scaling worker registry + heartbeats |
runtime_metadata | Per-tenant fan-out catch-up cursor |
SQL migrations run automatically at boot.
Environment variables (summary)
| Variable | Required | Default | Purpose |
|---|---|---|---|
DATABASE_URL | Yes | — | Postgres connection string |
YOKE_STORAGE | Yes | pg_sqlite (CLI) | pg_sqlite or s3 |
YOKE_TENANT_ID | No | default | Tenant scope for this process |
YOKE_DATA_DIR | No | .yoke-data | Local SQLite cache directory |
YOKE_PORT | No | 8080 | Portal HTTP port |
YOKE_WORKER | No | (runtime default) | Deno worker entry script |
YOKE_WORKER_PROJECT | No | (runtime default) | Deno project dir (deno.json) |
YOKE_WORKER_ID | No | {hostname}-{uuid} | Stable worker identity |
YOKE_MAX_QUEUE_DEPTH | No | 1000 | Per-instance queue cap |
YOKE_MAX_RETRIES | No | 5 | process() failures before dead letter |
YOKE_S3_BUCKET | S3 only | — | S3 bucket for actor SQLite |
YOKE_S3_PREFIX | No | "" | Key prefix (scoped under tenant_id) |
YOKE_S3_SYNC | No | lazy | lazy or strict local fsync before upload |
YOKE_EXECUTOR_MODE | No | — | 1 / true enables lease heartbeat |
YOKE_LEASE_HOLDER | No | — | Lease identity for hosted executors |
YOKE_LEASE_HEARTBEAT_INTERVAL_SECS | No | 15 | Executor lease tick interval |
See Storage and Deno worker for backend-specific details.
Pages in this section
Architecture
YokeRuntime components and data-flow diagram.
Stream
PgStream, sequencing, append path.
Fan-out
FanoutEngine, filters, idempotency.
Queues
actor_queues, priority, groups, retries.
Scheduler
ActorScheduler tick loop and Deno dispatch.
Storage
pg_sqlite vs S3, state CAS, store IDs.
Deno worker
EmbeddedDenoHost and JSON-RPC ops.
Boot sequence
Ordered steps in YokeRuntime::start.
Observability
GET /metrics on the portal HTTP port exposes Prometheus text format (alongside GET /health).
| Metric | Type | Description |
|---|---|---|
yoke_stream_appends_total | counter | Stream append operations |
yoke_fanout_enqueues_total | counter | Events enqueued via fan-out |
yoke_actor_process_total | counter | process() invocations |
yoke_actor_process_errors_total | counter | Failed process() calls |
yoke_queue_dead_letters_total | counter | Dead-lettered events |
yoke_queue_depth | gauge | Per-instance queue depth (instance_id label) |
yoke_runtime_workers | gauge | Workers with recent heartbeat |
Related docs
- Mental model — stream, actors, portals from the application side
- Mental model — design principles from the author perspective