Add bounded contextual entity resolution before interpretation

This commit is contained in:
Nico committed 2026-10-15 00:21:16 +02:00
1 parent 87581e5cd2
commit 4fa7fb6e74
21 files changed
+1166 -14

No files matched your search

+124
View File
@@ -0,0 +1,124 @@
# Contextual entity resolution
Entity resolution belongs to **Case Interpretation**, the first semantic phase.
The deterministic CaseBuilder and its completed `case.json` remain unchanged.
This step identifies entities needed to understand a report; it does not inspect
spawn availability, generate hypotheses, diagnose a cause or propose changes.
## Sequence
1. A versioned model prompt proposes grounded name searches, with contextual
roles and references to readable case sources. Search terms must occur in
the cited text; invented names, unknown catalogs and unread references fail
validation. The model cannot choose tables, columns, credentials or endpoints.
2. The controller validates the entire plan. The existing read-only project
research service executes parameterized, literal `contains` name queries
against operator-configured catalogs. Each result is captured in the package,
including its exact query, database snapshot binding and coverage limits.
3. A separate model assessment compares the returned candidates with the
original context. It returns `selected`, `ambiguous` or `no_match` for each
search. A selection must refer to an actual candidate from that search and
cite both its capture and the original mention. Truncated searches cannot
produce a unique selection. Similar names alone are insufficient.
4. The controller adds captured lookup facts and selected entities to the frozen
interpretation context. The final interpretation uses the resolved prompt.
Selected entities must be retained with mention citations; unresolved
identities must be retained as unknowns. These constraints participate in the
existing interpretation correction loop.
Normally this adds **two model calls** before the final interpretation. It is a
quality feature, not a token reduction. Invalid replies and provider retries can
add calls, within the persisted policies. No API request is needed to replay
completed resolution and interpretation artifacts.
## Configuration
`config/project.example.toml` demonstrates the optional `[entity_resolution]`
block. It is disabled when absent. The current Vcore example enables creature
name and localized creature name catalogs; the core mechanism is independent of
WoW and supports other entity types and database tables.
- `max_searches`: maximum proposed searches (default 4).
- `max_candidates`: maximum rows per search (default 8).
- `max_invalid_responses`: maximum invalid replies **per semantic stage**
(default 3).
- `catalogs`: operator-selected entity type, table, identity column, name column
and returned data columns. Tables must also be in the project DB allowlist.
These are configurable operational safety defaults, not package format limits.
No arbitrary SQL, regex, joins, model-supplied table names, network browsing,
runtime spawn counts or unlimited research loop is exposed here. Additional
catalogs can cover localized names or other entity domains. A miss is scoped to
the chosen catalog, name filter and configured snapshot; it never proves that
an entity is absent from the runtime world.
## Provenance and lifecycle
Artifacts live below
`analysis/interpretations/NNNN/entity-resolution/`: frozen input and policy,
accepted plan, content-addressed query captures, candidate index, assessment,
and attempt-specific rejected response records. The usual LLM artifacts retain
provider, model, token counts, timing and validated requests/responses for both
semantic stages. Every accepted stage is checkpointed.
Existing canonical source aliases stay unchanged. New `F` aliases expose captured
technical lookup results; new `N` aliases expose selected entities with exact
capture row provenance, mention references and an assessment reference. New `D`
aliases expose unresolved identities. Their `/entity_resolution/` pointers refer
to these separate artifacts, not invented fields inside the canonical case.
A DB-backed ID/name is a technical fact. Its assigned role is explicitly labeled
`model_contextual_selection_not_causation`: semantic association is not a proof
of runtime behavior or of a defect. An existing quest giver remains relevant and
can still be a contributor; it is not equated with the missing target creature.
Not seeing a creature is not automatically proof of missing spawn records.
Restart reuses accepted plans, captures and assessments. A database outage blocks
the case as `RESEARCH_UNAVAILABLE` without losing the plan. Invalid response
exhaustion blocks as `ENTITY_RESOLUTION_INVALID_RESPONSES`. Neither changes
hypothesis budgets or creates hypotheses. Blocked cases do not hold a worker;
resumption is explicit, after the recorded retry timestamp (60-second safety
delay). Exhausted response policies require review and a new analysis revision.
Changing catalogs or snapshot bindings during an unfinished stage is rejected.
Completed results can be verified in a different workspace without a live DB.
Existing completed/started interpretations are not silently enriched or rewritten.
They retain their original prompts and contexts; use a fresh case or a new
analysis revision to test this feature. `--stop-after interpretation` includes
resolution and still prevents hypothesis generation and investigation.
## Tests
Offline tests cover unobserved affected subjects, competing candidates, unknowns,
grounding, limits, invalid response corrections/exhaustion, scoped empty results,
DB outages, provider retries, changed bindings, append-only preservation,
portable replay and the trial phase stop.
Opt-in tests use `VBA_TEST_PROJECT_CONFIG` for the local SELECT-only snapshot.
With `VBA_TEST_CASE_PACKAGE` pointing to a built but not yet interpreted package,
an additional end-to-end test uses the real case and database, with **fixture
model replies**. It proves query/provenance/lifecycle integration, not the quality
of a live model's semantic decisions. For the known Wolpertinger example, the
fixture proposes the name found in the quest text; the real DB supplies the ID.
The implementation contains no case-specific Creature-ID or diagnosis.
## Prepared development trial
A fresh copy of BUG-000020 is prepared in
`service-data/trials/entity-resolution-20261015`, package ID
`trial-entity-resolution-BUG-000020-20261015`. Only import, admission and
deterministic case construction have run (generation 4, no model calls).
Run from the repository root in the PowerShell session containing the API key:
```powershell
.\.venv\Scripts\python.exe -m viano_bug_analysis.trial run --workspace service-data/trials/entity-resolution-20261015 --llm-config config/llm.timeout300.local.toml --project-config config/project.local.toml --case-id BUG-000020 --package-id trial-entity-resolution-BUG-000020-20261015 --stop-after interpretation --max-dispatches 6
```
This makes billable model requests but stops before hypotheses/investigation.
If a provider retry is pending, repeat the same command after its timestamp.
After completion, inspect the resolution plan, captured candidates, assessment
and final interpretation together. Success means the model itself proposes a
context-grounded search, distinguishes Goldark from the affected creature, and
retains the selected identity and its provenance. A fixture test alone does not
establish this live-model quality.