# LLMGateway v1

This milestone implements architecture section 13: configuration, provider transport,
ordered failover, bounded retries, schema validation, metadata and lifecycle checkpoints.
The gateway itself does not interpret cases, generate hypotheses or execute Investigation
Tools. The separate CaseInterpreter now supplies the first domain prompt and schema
(see case-interpretation-v1.md). The existing CaseBuilder remains deterministic and does
not start an LLM request on its own.

## Configuration

Copy `config/llm.example.toml` to `config/llm.local.toml` (ignored by Git). Provider array
order is selection order; disabled providers are skipped. URL, model and adapter are
independent. The bundled `openai_compatible` adapter uses a base URL ending in `/v1`
and appends `/chat/completions`. An own server or Colab-hosted service exposing that
protocol needs only another configuration entry. Other protocols can implement the
async `LLMProvider` interface and register an adapter with `LLMGateway(..., adapters=...)`.
There is no provider-specific behavior in the runner or case-processing modules.

Keys are referenced by `api_key_env`; actual values come from process environment
variables. Inline credentials and URL credentials are not supported. Neither resolved
keys, response error bodies nor endpoint URLs are included in logs/checkpoints.
Nonlocal HTTP requires explicit `allow_insecure_http = true`; HTTPS verifies certificates.
Redirects are rejected and ambient proxy settings are not inherited.

`privacy_profile` determines the allowed destination classes:

| Profile | Eligible providers |
| --- | --- |
| `local_only` (default) | loopback/local |
| `self_hosted` | local and explicitly trusted self-hosted |
| `cloud_allowed` | local, self-hosted and cloud |

The gateway preserves configured order within the eligible set. A request may narrow
that policy but cannot broaden it. PromptBuilder defaults to the configured policy.
To enable cloud failover, enable the cloud entry and set `privacy_profile = "cloud_allowed"`.
Missing credentials or having no eligible provider is a configuration error.

`response_format` may be `json_schema`, `json_object` or `none`, accommodating different
server capabilities. JSON is always required and validated locally against the caller's
schema. `token_parameter` selects `max_completion_tokens` or `max_tokens`. No unsupported
parameter retry or hidden downgrade occurs. The Chat Completions transport follows the
[official OpenAI API reference](https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create).

## Failover and retry semantics

One attempt is one complete round through eligible providers. Each round starts with
the first provider, immediately proceeds to the next on connection failures, timeout,
HTTP 408/429 or HTTP 5xx, and stops on the first validated success. Authentication errors,
other HTTP configuration failures, refusals, truncated model output, invalid JSON and
schema violations are controlled errors; they are not logged as unreachable servers.
Adapters execute one call and never retry independently.

HTTP failures additionally preserve an optional numeric `http_status` (100–599).
Temporary failures record it in `outcome.attempts` or successful failover metadata;
nonretryable HTTP errors record it in `outcome.http_status`. Existing v1 records
without this field remain valid. No provider response body, headers or credentials
are retained. Retry/failover classification is unchanged: 408/429/5xx are temporary;
a 429 alone does not distinguish rate limits from exhausted quota.

Operational defaults:

| Setting | Default |
| --- | --- |
| `max_attempts` | 10 full rounds |
| `retry_delay_seconds` | 60 between failed rounds |
| `blocked_retry_seconds` | 3600 before an explicit restart of an exhausted operation |
| provider `timeout_seconds` | 30 |
| `max_request_bytes` / `max_response_bytes` | 1 MiB each |

These are configurable safety defaults, not immutable package-format constraints.
For N eligible providers, an unsuccessful operation makes at most 10 × N calls,
excluding a possible replay of the last unconfirmed round after a crash.

`LLMGateway.complete_round()` is an async transport operation. `CaseLLMRunner.run()`
executes at most one due round per dispatch. After an unsuccessful round it persists
`RETRY_PENDING` with a due timestamp and returns immediately. A scheduler can dispatch
other cases while waiting and invoke the same operation again when due. Calling early
returns `DEFERRED`, with no request and no new package generation. This milestone adds
the runner API, not a continuously running worker daemon or a domain-phase controller.

After the configured number of rounds, the runner commits `BLOCKED`, reason
`LLM_UNAVAILABLE`, destination `inbox`, a retry timestamp and `automatic_retry: false`.
The managed package remains in cases; routing is metadata. The InboxProcessor respects
that flag, including after a blocked package is received by another service. An exhausted
operation does not silently start another ten-round campaign. An explicit restart needs
resumption through CheckpointStore after the due date and a new stable operation_id.
Other BLOCKED cases retain their existing automatic retry behavior.

## Durable state and integration

Each operation_id is stable within an analysis revision. Checkpoints contain request
and configuration fingerprints, round count, phase, due time and immutable artifact
references. A changed request/configuration under the same operation_id is rejected;
use a new operation_id for deliberately changed configuration or input.

Under `llm/operations/<hashed revision and operation id>/`, the package contains the
exact credential-free request and one immutable outcome file per completed round.
Outcome files contain provider/model/prompt version, available token usage, elapsed
time and earlier failed providers in that round. Optional configured input/output rates
produce an estimated cost in `cost_currency`; missing usage/rates yield null, never a
guessed zero. Failed/unconfirmed calls may incur costs that cannot be measured locally.
Persisted files are covered by package integrity. Generated LLM artifacts do not become
raw CaseBuilder inputs. The original package/files and package_id remain unchanged;
package_generation increases per checkpoint, analysis_revision does not.

A confirmed success or controlled error is reused after restart without another model
call. Dispatches for the same case use a nonblocking operation lock; they report LockBusy
instead of waiting. The external request never holds the global worker lock. Concurrent
cases remain independent. Checkpoint commit uses the expected generation to reject stale
results prepared while another component modified the case. A crash may repeat the last
unconfirmed remote call; remote execution and local ZIP publication cannot be atomic.

The runner records failure logs in `runtime/logs/llm.jsonl` (rotating at 1 MiB with three
backups). An unreachable-server failure during BUILDING_CASE explicitly identifies
Casebuilding; another phase is logged by its actual name. The gateway logs only provider
IDs and safe error codes, not prompts, raw provider error bodies or credentials.

The generic versioned prompt `structured_json_v1.txt` prepares JSON transport and treats
input as untrusted data. It does not interpret a case or provide tools. Caller-supplied
output schemas cannot resolve external references; local JSON Schema and timestamp
validation runs before accepting any output.

The stdlib HTTP adapter runs blocking I/O in an executor. Async deadlines release the
awaiting case; cancellation cannot forcibly terminate an OS DNS lookup or already
running remote inference. A late result is discarded. Providers must respect transport
timeouts; a future native async adapter can implement the same interface without
changing configuration/lifecycle contracts.

Example integration after an active phase has been established:

```python
config = GatewayConfig.load("config/llm.local.toml")
gateway = LLMGateway(config)
runner = CaseLLMRunner(store, gateway)
request = PromptBuilder().build(
    task="Transport test",
    data={"observation": "A supplied observation"},
    output_schema={"type": "object", "properties": {"answer": {"type": "string"}},
                   "required": ["answer"], "additionalProperties": False},
)
result = await runner.run(case_id, package_id, operation_id="transport-test-v1", request=request)
# SUCCESS / RETRY_PENDING / DEFERRED / BLOCKED / ERROR
```

Tests use deterministic fake adapters and an actual local HTTP server. No cloud
credentials, paid calls or real models are required. CaseInterpreter uses this gateway
with its own versioned prompt and output schema.