Inference

defineInferenceStack, LLM providers, model routing, failover, and ctx.llm.infer.

Yoke's inference layer provides a canonical LLM interface for processor actors. All providers accept and return InferenceMessage[] using the same body.parts schema as the conversation store.

InferenceMessage

type InferenceRole = "user" | "agent";

type InferenceMessage = {
  type: string;
  role: InferenceRole;
  body: MessageBody;  // { parts[], meta? }
};

Roles are derived at inference time (not stored on messages):

Source	Role
External shadow actors (ingested forum authors)	`user`
`tool_result` parts	`user`
`tool_call` / `reasoning` / `refusal`	`agent`
Otherwise	Compare author participant → agent actor id

defineInferenceStack

import {
  defineInferenceStack,
  createOpenAiProvider,
  createAnthropicProvider,
  createFixtureProvider,
  defineRuntime,
} from "@yoke/sdk";

const inferenceStack = defineInferenceStack({
  models: {
    "gpt-4o": { id: "gpt-4o", contextWindow: 128_000 },
    "claude-sonnet-4": { id: "claude-sonnet-4", contextWindow: 200_000 },
    "fixture-echo": { id: "fixture-echo" },
  },
  providers: {
    fixture: createFixtureProvider(),
    openai: createOpenAiProvider({
      apiKey: Deno.env.get("OPENAI_API_KEY") ?? "",
    }),
    anthropic: createAnthropicProvider({
      apiKey: Deno.env.get("ANTHROPIC_API_KEY") ?? "",
    }),
  },
  routing: {
    "gpt-4o": ["openai", "fixture"],
    "claude-sonnet-4": ["anthropic"],
    "fixture-echo": ["fixture"],
  },
});

export default defineRuntime()
  .inferenceStack(inferenceStack)
  .actors(/* ... */);

Without .inferenceStack(), ctx.llm.infer() throws at runtime.

ctx.llm.infer

Inside processor process():

const outputs = await ctx.llm.infer(inputMessages, {
  model: ctx.config.model ?? "fixture-echo",
  temperature: 0.7,
  maxTokens: 1024,
  tools: resolvedTools,
});

Option	Purpose
`model`	Logical model name from stack
`temperature`	Sampling temperature
`maxTokens`	Output token limit
`tools`	Resolved tools/toolchains for this round

Returns InferenceMessage[] to post into the monologue.

Router behavior

The inference router handles:

Provider selection — first available in routing array for model
Round-robin — load balance across provider instances
Retries — transient failures
Failover — next provider in routing list (e.g. ["openai", "fixture"])

Providers

Factory	Env	Use
`createOpenAiProvider`	`OPENAI_API_KEY`	GPT models
`createAnthropicProvider`	`ANTHROPIC_API_KEY`	Claude models
`createFixtureProvider`	none	Tests, CI
`createLocalToolLoopProvider`	none	Local tool demo loop
`createSaysV1TestProvider`	none	Deterministic test replies
`createGuardianSteerTestProvider`	none	Guardian angel E2E

Routing example

routing: {
  "local-tool-loop": ["local"],
  "fixture-echo": ["fixture"],
  "gpt-4o": ["openai", "fixture"],
  "claude-sonnet-4": ["anthropic"],
  "guardian-steer": ["guardian-steer"],
},

Join conversation with model: "local-tool-loop" for offline demos.

Processor integration

The processor reads monologue transcript → converts to InferenceMessage[] → optionally compacts → calls ctx.llm.infer → posts output parts to monologue.

One infer step per conversation.message.posted event — not a blocking while-loop. See Processor.

Tools

Tool registration for infer.

Compaction

Pre-infer transcript compaction.

Message types

body.parts schema.

Tools

Compaction

Message types

On this page