8.6 KiB
LLMGateway v1
This milestone implements architecture section 13: configuration, provider transport, ordered failover, bounded retries, schema validation, metadata and lifecycle checkpoints. The gateway itself does not interpret cases, generate hypotheses or execute Investigation Tools. The separate CaseInterpreter now supplies the first domain prompt and schema (see case-interpretation-v1.md). The existing CaseBuilder remains deterministic and does not start an LLM request on its own.
Configuration
Copy config/llm.example.toml to config/llm.local.toml (ignored by Git). Provider array
order is selection order; disabled providers are skipped. URL, model and adapter are
independent. The bundled openai_compatible adapter uses a base URL ending in /v1
and appends /chat/completions. An own server or Colab-hosted service exposing that
protocol needs only another configuration entry. Other protocols can implement the
async LLMProvider interface and register an adapter with LLMGateway(..., adapters=...).
There is no provider-specific behavior in the runner or case-processing modules.
Keys are referenced by api_key_env; actual values come from process environment
variables. Inline credentials and URL credentials are not supported. Neither resolved
keys, response error bodies nor endpoint URLs are included in logs/checkpoints.
Nonlocal HTTP requires explicit allow_insecure_http = true; HTTPS verifies certificates.
Redirects are rejected and ambient proxy settings are not inherited.
privacy_profile determines the allowed destination classes:
| Profile | Eligible providers |
|---|---|
local_only (default) |
loopback/local |
self_hosted |
local and explicitly trusted self-hosted |
cloud_allowed |
local, self-hosted and cloud |
The gateway preserves configured order within the eligible set. A request may narrow
that policy but cannot broaden it. PromptBuilder defaults to the configured policy.
To enable cloud failover, enable the cloud entry and set privacy_profile = "cloud_allowed".
Missing credentials or having no eligible provider is a configuration error.
response_format may be json_schema, json_object or none, accommodating different
server capabilities. JSON is always required and validated locally against the caller's
schema. token_parameter selects max_completion_tokens or max_tokens. No unsupported
parameter retry or hidden downgrade occurs. The Chat Completions transport follows the
official OpenAI API reference.
Failover and retry semantics
One attempt is one complete round through eligible providers. Each round starts with the first provider, immediately proceeds to the next on connection failures, timeout, HTTP 408/429 or HTTP 5xx, and stops on the first validated success. Authentication errors, other HTTP configuration failures, refusals, truncated model output, invalid JSON and schema violations are controlled errors; they are not logged as unreachable servers. Adapters execute one call and never retry independently.
HTTP failures additionally preserve an optional numeric http_status (100–599).
Temporary failures record it in outcome.attempts or successful failover metadata;
nonretryable HTTP errors record it in outcome.http_status. Existing v1 records
without this field remain valid. No provider response body, headers or credentials
are retained. Retry/failover classification is unchanged: 408/429/5xx are temporary;
a 429 alone does not distinguish rate limits from exhausted quota.
Operational defaults:
| Setting | Default |
|---|---|
max_attempts |
10 full rounds |
retry_delay_seconds |
60 between failed rounds |
blocked_retry_seconds |
3600 before an explicit restart of an exhausted operation |
provider timeout_seconds |
30 |
max_request_bytes / max_response_bytes |
1 MiB each |
These are configurable safety defaults, not immutable package-format constraints. For N eligible providers, an unsuccessful operation makes at most 10 × N calls, excluding a possible replay of the last unconfirmed round after a crash.
LLMGateway.complete_round() is an async transport operation. CaseLLMRunner.run()
executes at most one due round per dispatch. After an unsuccessful round it persists
RETRY_PENDING with a due timestamp and returns immediately. A scheduler can dispatch
other cases while waiting and invoke the same operation again when due. Calling early
returns DEFERRED, with no request and no new package generation. This milestone adds
the runner API, not a continuously running worker daemon or a domain-phase controller.
After the configured number of rounds, the runner commits BLOCKED, reason
LLM_UNAVAILABLE, destination inbox, a retry timestamp and automatic_retry: false.
The managed package remains in cases; routing is metadata. The InboxProcessor respects
that flag, including after a blocked package is received by another service. An exhausted
operation does not silently start another ten-round campaign. An explicit restart needs
resumption through CheckpointStore after the due date and a new stable operation_id.
Other BLOCKED cases retain their existing automatic retry behavior.
Durable state and integration
Each operation_id is stable within an analysis revision. Checkpoints contain request and configuration fingerprints, round count, phase, due time and immutable artifact references. A changed request/configuration under the same operation_id is rejected; use a new operation_id for deliberately changed configuration or input.
Under llm/operations/<hashed revision and operation id>/, the package contains the
exact credential-free request and one immutable outcome file per completed round.
Outcome files contain provider/model/prompt version, available token usage, elapsed
time and earlier failed providers in that round. Optional configured input/output rates
produce an estimated cost in cost_currency; missing usage/rates yield null, never a
guessed zero. Failed/unconfirmed calls may incur costs that cannot be measured locally.
Persisted files are covered by package integrity. Generated LLM artifacts do not become
raw CaseBuilder inputs. The original package/files and package_id remain unchanged;
package_generation increases per checkpoint, analysis_revision does not.
A confirmed success or controlled error is reused after restart without another model call. Dispatches for the same case use a nonblocking operation lock; they report LockBusy instead of waiting. The external request never holds the global worker lock. Concurrent cases remain independent. Checkpoint commit uses the expected generation to reject stale results prepared while another component modified the case. A crash may repeat the last unconfirmed remote call; remote execution and local ZIP publication cannot be atomic.
The runner records failure logs in runtime/logs/llm.jsonl (rotating at 1 MiB with three
backups). An unreachable-server failure during BUILDING_CASE explicitly identifies
Casebuilding; another phase is logged by its actual name. The gateway logs only provider
IDs and safe error codes, not prompts, raw provider error bodies or credentials.
The generic versioned prompt structured_json_v1.txt prepares JSON transport and treats
input as untrusted data. It does not interpret a case or provide tools. Caller-supplied
output schemas cannot resolve external references; local JSON Schema and timestamp
validation runs before accepting any output.
The stdlib HTTP adapter runs blocking I/O in an executor. Async deadlines release the awaiting case; cancellation cannot forcibly terminate an OS DNS lookup or already running remote inference. A late result is discarded. Providers must respect transport timeouts; a future native async adapter can implement the same interface without changing configuration/lifecycle contracts.
Example integration after an active phase has been established:
config = GatewayConfig.load("config/llm.local.toml")
gateway = LLMGateway(config)
runner = CaseLLMRunner(store, gateway)
request = PromptBuilder().build(
task="Transport test",
data={"observation": "A supplied observation"},
output_schema={"type": "object", "properties": {"answer": {"type": "string"}},
"required": ["answer"], "additionalProperties": False},
)
result = await runner.run(case_id, package_id, operation_id="transport-test-v1", request=request)
# SUCCESS / RETRY_PENDING / DEFERRED / BLOCKED / ERROR
Tests use deterministic fake adapters and an actual local HTTP server. No cloud credentials, paid calls or real models are required. CaseInterpreter uses this gateway with its own versioned prompt and output schema.