149 lines
8.6 KiB
Markdown
149 lines
8.6 KiB
Markdown
# LLMGateway v1
|
||
|
||
This milestone implements architecture section 13: configuration, provider transport,
|
||
ordered failover, bounded retries, schema validation, metadata and lifecycle checkpoints.
|
||
The gateway itself does not interpret cases, generate hypotheses or execute Investigation
|
||
Tools. The separate CaseInterpreter now supplies the first domain prompt and schema
|
||
(see case-interpretation-v1.md). The existing CaseBuilder remains deterministic and does
|
||
not start an LLM request on its own.
|
||
|
||
## Configuration
|
||
|
||
Copy `config/llm.example.toml` to `config/llm.local.toml` (ignored by Git). Provider array
|
||
order is selection order; disabled providers are skipped. URL, model and adapter are
|
||
independent. The bundled `openai_compatible` adapter uses a base URL ending in `/v1`
|
||
and appends `/chat/completions`. An own server or Colab-hosted service exposing that
|
||
protocol needs only another configuration entry. Other protocols can implement the
|
||
async `LLMProvider` interface and register an adapter with `LLMGateway(..., adapters=...)`.
|
||
There is no provider-specific behavior in the runner or case-processing modules.
|
||
|
||
Keys are referenced by `api_key_env`; actual values come from process environment
|
||
variables. Inline credentials and URL credentials are not supported. Neither resolved
|
||
keys, response error bodies nor endpoint URLs are included in logs/checkpoints.
|
||
Nonlocal HTTP requires explicit `allow_insecure_http = true`; HTTPS verifies certificates.
|
||
Redirects are rejected and ambient proxy settings are not inherited.
|
||
|
||
`privacy_profile` determines the allowed destination classes:
|
||
|
||
| Profile | Eligible providers |
|
||
| --- | --- |
|
||
| `local_only` (default) | loopback/local |
|
||
| `self_hosted` | local and explicitly trusted self-hosted |
|
||
| `cloud_allowed` | local, self-hosted and cloud |
|
||
|
||
The gateway preserves configured order within the eligible set. A request may narrow
|
||
that policy but cannot broaden it. PromptBuilder defaults to the configured policy.
|
||
To enable cloud failover, enable the cloud entry and set `privacy_profile = "cloud_allowed"`.
|
||
Missing credentials or having no eligible provider is a configuration error.
|
||
|
||
`response_format` may be `json_schema`, `json_object` or `none`, accommodating different
|
||
server capabilities. JSON is always required and validated locally against the caller's
|
||
schema. `token_parameter` selects `max_completion_tokens` or `max_tokens`. No unsupported
|
||
parameter retry or hidden downgrade occurs. The Chat Completions transport follows the
|
||
[official OpenAI API reference](https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create).
|
||
|
||
## Failover and retry semantics
|
||
|
||
One attempt is one complete round through eligible providers. Each round starts with
|
||
the first provider, immediately proceeds to the next on connection failures, timeout,
|
||
HTTP 408/429 or HTTP 5xx, and stops on the first validated success. Authentication errors,
|
||
other HTTP configuration failures, refusals, truncated model output, invalid JSON and
|
||
schema violations are controlled errors; they are not logged as unreachable servers.
|
||
Adapters execute one call and never retry independently.
|
||
|
||
HTTP failures additionally preserve an optional numeric `http_status` (100–599).
|
||
Temporary failures record it in `outcome.attempts` or successful failover metadata;
|
||
nonretryable HTTP errors record it in `outcome.http_status`. Existing v1 records
|
||
without this field remain valid. No provider response body, headers or credentials
|
||
are retained. Retry/failover classification is unchanged: 408/429/5xx are temporary;
|
||
a 429 alone does not distinguish rate limits from exhausted quota.
|
||
|
||
Operational defaults:
|
||
|
||
| Setting | Default |
|
||
| --- | --- |
|
||
| `max_attempts` | 10 full rounds |
|
||
| `retry_delay_seconds` | 60 between failed rounds |
|
||
| `blocked_retry_seconds` | 3600 before an explicit restart of an exhausted operation |
|
||
| provider `timeout_seconds` | 30 |
|
||
| `max_request_bytes` / `max_response_bytes` | 1 MiB each |
|
||
|
||
These are configurable safety defaults, not immutable package-format constraints.
|
||
For N eligible providers, an unsuccessful operation makes at most 10 × N calls,
|
||
excluding a possible replay of the last unconfirmed round after a crash.
|
||
|
||
`LLMGateway.complete_round()` is an async transport operation. `CaseLLMRunner.run()`
|
||
executes at most one due round per dispatch. After an unsuccessful round it persists
|
||
`RETRY_PENDING` with a due timestamp and returns immediately. A scheduler can dispatch
|
||
other cases while waiting and invoke the same operation again when due. Calling early
|
||
returns `DEFERRED`, with no request and no new package generation. This milestone adds
|
||
the runner API, not a continuously running worker daemon or a domain-phase controller.
|
||
|
||
After the configured number of rounds, the runner commits `BLOCKED`, reason
|
||
`LLM_UNAVAILABLE`, destination `inbox`, a retry timestamp and `automatic_retry: false`.
|
||
The managed package remains in cases; routing is metadata. The InboxProcessor respects
|
||
that flag, including after a blocked package is received by another service. An exhausted
|
||
operation does not silently start another ten-round campaign. An explicit restart needs
|
||
resumption through CheckpointStore after the due date and a new stable operation_id.
|
||
Other BLOCKED cases retain their existing automatic retry behavior.
|
||
|
||
## Durable state and integration
|
||
|
||
Each operation_id is stable within an analysis revision. Checkpoints contain request
|
||
and configuration fingerprints, round count, phase, due time and immutable artifact
|
||
references. A changed request/configuration under the same operation_id is rejected;
|
||
use a new operation_id for deliberately changed configuration or input.
|
||
|
||
Under `llm/operations/<hashed revision and operation id>/`, the package contains the
|
||
exact credential-free request and one immutable outcome file per completed round.
|
||
Outcome files contain provider/model/prompt version, available token usage, elapsed
|
||
time and earlier failed providers in that round. Optional configured input/output rates
|
||
produce an estimated cost in `cost_currency`; missing usage/rates yield null, never a
|
||
guessed zero. Failed/unconfirmed calls may incur costs that cannot be measured locally.
|
||
Persisted files are covered by package integrity. Generated LLM artifacts do not become
|
||
raw CaseBuilder inputs. The original package/files and package_id remain unchanged;
|
||
package_generation increases per checkpoint, analysis_revision does not.
|
||
|
||
A confirmed success or controlled error is reused after restart without another model
|
||
call. Dispatches for the same case use a nonblocking operation lock; they report LockBusy
|
||
instead of waiting. The external request never holds the global worker lock. Concurrent
|
||
cases remain independent. Checkpoint commit uses the expected generation to reject stale
|
||
results prepared while another component modified the case. A crash may repeat the last
|
||
unconfirmed remote call; remote execution and local ZIP publication cannot be atomic.
|
||
|
||
The runner records failure logs in `runtime/logs/llm.jsonl` (rotating at 1 MiB with three
|
||
backups). An unreachable-server failure during BUILDING_CASE explicitly identifies
|
||
Casebuilding; another phase is logged by its actual name. The gateway logs only provider
|
||
IDs and safe error codes, not prompts, raw provider error bodies or credentials.
|
||
|
||
The generic versioned prompt `structured_json_v1.txt` prepares JSON transport and treats
|
||
input as untrusted data. It does not interpret a case or provide tools. Caller-supplied
|
||
output schemas cannot resolve external references; local JSON Schema and timestamp
|
||
validation runs before accepting any output.
|
||
|
||
The stdlib HTTP adapter runs blocking I/O in an executor. Async deadlines release the
|
||
awaiting case; cancellation cannot forcibly terminate an OS DNS lookup or already
|
||
running remote inference. A late result is discarded. Providers must respect transport
|
||
timeouts; a future native async adapter can implement the same interface without
|
||
changing configuration/lifecycle contracts.
|
||
|
||
Example integration after an active phase has been established:
|
||
|
||
```python
|
||
config = GatewayConfig.load("config/llm.local.toml")
|
||
gateway = LLMGateway(config)
|
||
runner = CaseLLMRunner(store, gateway)
|
||
request = PromptBuilder().build(
|
||
task="Transport test",
|
||
data={"observation": "A supplied observation"},
|
||
output_schema={"type": "object", "properties": {"answer": {"type": "string"}},
|
||
"required": ["answer"], "additionalProperties": False},
|
||
)
|
||
result = await runner.run(case_id, package_id, operation_id="transport-test-v1", request=request)
|
||
# SUCCESS / RETRY_PENDING / DEFERRED / BLOCKED / ERROR
|
||
```
|
||
|
||
Tests use deterministic fake adapters and an actual local HTTP server. No cloud
|
||
credentials, paid calls or real models are required. CaseInterpreter uses this gateway
|
||
with its own versioned prompt and output schema.
|