219 lines
13 KiB
Markdown
219 lines
13 KiB
Markdown
# Contextual entity resolution
|
|
|
|
Entity resolution belongs to **Case Interpretation**, the first semantic phase.
|
|
The deterministic CaseBuilder and its completed `case.json` remain unchanged.
|
|
This step identifies entities needed to understand a report; it does not inspect
|
|
spawn availability, generate hypotheses, diagnose a cause or propose changes.
|
|
|
|
## Sequence
|
|
|
|
1. A versioned model prompt proposes grounded name searches, with contextual
|
|
roles and references to readable case sources. Search terms must occur in
|
|
the cited text; invented names, unknown catalogs and unread references fail
|
|
validation. The model cannot choose tables, columns, credentials or endpoints.
|
|
2. The controller validates the entire plan. The existing read-only project
|
|
research service executes parameterized, literal `contains` name queries
|
|
against operator-configured catalogs. Each result is captured in the package,
|
|
including its exact query, database snapshot binding and coverage limits.
|
|
3. A separate model assessment compares the returned candidates with the
|
|
original context. It returns `selected`, `ambiguous` or `no_match` for each
|
|
search. A selection must refer to an actual candidate from that search and
|
|
cite both its capture and the original mention. Truncated searches cannot
|
|
produce a unique selection. Similar names alone are insufficient.
|
|
4. The controller adds captured lookup facts and selected entities to the frozen
|
|
interpretation context. The final interpretation uses the resolved prompt.
|
|
Selected entities must be retained with mention citations; unresolved
|
|
identities must be retained as unknowns. These constraints participate in the
|
|
existing interpretation correction loop.
|
|
|
|
Normally this adds **two model calls** before the final interpretation. It is a
|
|
quality feature, not a token reduction. Invalid replies and provider retries can
|
|
add calls, within the persisted policies. No API request is needed to replay
|
|
completed resolution and interpretation artifacts.
|
|
|
|
## Configuration
|
|
|
|
`config/project.example.toml` demonstrates the optional `[entity_resolution]`
|
|
block. It is disabled when absent. The current Vcore example enables creature
|
|
name and localized creature name catalogs; the core mechanism is independent of
|
|
WoW and supports other entity types and database tables.
|
|
|
|
- `max_searches`: maximum proposed searches (default 4).
|
|
- `max_candidates`: maximum rows per search (default 8).
|
|
- `max_invalid_responses`: maximum invalid replies **per semantic stage**
|
|
(default 3).
|
|
- `catalogs`: operator-selected entity type, table, identity column, name column
|
|
and returned data columns. Tables must also be in the project DB allowlist.
|
|
|
|
These are configurable operational safety defaults, not package format limits.
|
|
No arbitrary SQL, regex, joins, model-supplied table names, network browsing,
|
|
runtime spawn counts or unlimited research loop is exposed here. Additional
|
|
catalogs can cover localized names or other entity domains. A miss is scoped to
|
|
the chosen catalog, name filter and configured snapshot; it never proves that
|
|
an entity is absent from the runtime world.
|
|
|
|
## Provenance and lifecycle
|
|
|
|
Artifacts live below
|
|
`analysis/interpretations/NNNN/entity-resolution/`: frozen input and policy,
|
|
accepted plan, content-addressed query captures, candidate index, assessment,
|
|
and attempt-specific rejected response records. The usual LLM artifacts retain
|
|
provider, model, token counts, timing and validated requests/responses for both
|
|
semantic stages. Every accepted stage is checkpointed.
|
|
|
|
Existing canonical source aliases stay unchanged. New `F` aliases expose captured
|
|
technical lookup results; new `N` aliases expose selected entities with exact
|
|
capture row provenance, mention references and an assessment reference. New `D`
|
|
aliases expose unresolved identities. Their `/entity_resolution/` pointers refer
|
|
to these separate artifacts, not invented fields inside the canonical case.
|
|
|
|
A DB-backed ID/name is a technical fact. Its assigned role is explicitly labeled
|
|
`model_contextual_selection_not_causation`: semantic association is not a proof
|
|
of runtime behavior or of a defect. An existing quest giver remains relevant and
|
|
can still be a contributor; it is not equated with the missing target creature.
|
|
Not seeing a creature is not automatically proof of missing spawn records.
|
|
|
|
Restart reuses accepted plans, captures and assessments. A database outage blocks
|
|
the case as `RESEARCH_UNAVAILABLE` without losing the plan. Invalid response
|
|
exhaustion blocks as `ENTITY_RESOLUTION_INVALID_RESPONSES`. Neither changes
|
|
hypothesis budgets or creates hypotheses. Blocked cases do not hold a worker;
|
|
resumption is explicit, after the recorded retry timestamp (60-second safety
|
|
delay). Exhausted response policies require review and a new analysis revision.
|
|
Changing catalogs or snapshot bindings during an unfinished stage is rejected.
|
|
Completed results can be verified in a different workspace without a live DB.
|
|
|
|
Existing completed/started interpretations are not silently enriched or rewritten.
|
|
They retain their original prompts and contexts; use a fresh case or a new
|
|
analysis revision to test this feature. `--stop-after interpretation` includes
|
|
resolution and still prevents hypothesis generation and investigation.
|
|
|
|
## Tests
|
|
|
|
Offline tests cover unobserved affected subjects, competing candidates, unknowns,
|
|
grounding, limits, invalid response corrections/exhaustion, scoped empty results,
|
|
DB outages, provider retries, changed bindings, append-only preservation,
|
|
portable replay and the trial phase stop.
|
|
|
|
Opt-in tests use `VBA_TEST_PROJECT_CONFIG` for the local SELECT-only snapshot.
|
|
With `VBA_TEST_CASE_PACKAGE` pointing to a built but not yet interpreted package,
|
|
an additional end-to-end test uses the real case and database, with **fixture
|
|
model replies**. It proves query/provenance/lifecycle integration, not the quality
|
|
of a live model's semantic decisions. For the known Wolpertinger example, the
|
|
fixture proposes the name found in the quest text; the real DB supplies the ID.
|
|
The implementation contains no case-specific Creature-ID or diagnosis.
|
|
|
|
## Prepared development trial
|
|
|
|
A fresh copy of BUG-000020 is prepared in
|
|
`service-data/trials/entity-resolution-20261015`, package ID
|
|
`trial-entity-resolution-BUG-000020-20261015`. Only import, admission and
|
|
deterministic case construction have run (generation 4, no model calls).
|
|
|
|
Run from the repository root in the PowerShell session containing the API key:
|
|
|
|
```powershell
|
|
.\.venv\Scripts\python.exe -m viano_bug_analysis.trial run --workspace service-data/trials/entity-resolution-20261015 --llm-config config/llm.timeout300.local.toml --project-config config/project.local.toml --case-id BUG-000020 --package-id trial-entity-resolution-BUG-000020-20261015 --stop-after interpretation --max-dispatches 6
|
|
```
|
|
|
|
This makes billable model requests but stops before hypotheses/investigation.
|
|
If a provider retry is pending, repeat the same command after its timestamp.
|
|
After completion, inspect the resolution plan, captured candidates, assessment
|
|
and final interpretation together. Success means the model itself proposes a
|
|
context-grounded search, distinguishes Goldark from the affected creature, and
|
|
retains the selected identity and its provenance. A fixture test alone does not
|
|
establish this live-model quality.
|
|
|
|
## Schema visibility fix (prompt protocol v2)
|
|
|
|
New resolution runs freeze `protocol_version = 2` in their input artifact. Both
|
|
planning and assessment include the output schema in the user message, so
|
|
providers configured with `response_format = "json_object"` can see the required
|
|
fields. That provider mode only requests JSON; it does not transmit the internal
|
|
request schema. Protocol v1 omitted that message field. Existing v1 artifacts
|
|
retain their original request construction and prompts for immutable replay.
|
|
|
|
The first live entity-resolution trial exhausted planning validation: two
|
|
`LLM_OUTPUT_SCHEMA` results and one `LLM_INCOMPLETE_OUTPUT`. No lookup was
|
|
executed. The error records do not retain rejected response bodies or the exact
|
|
incomplete finish reason, so they cannot establish each response's detailed
|
|
cause. Do not retry that exhausted case unchanged; use the fresh trial below.
|
|
|
|
A corrected trial is prepared at `service-data/trials/entity-resolution-schema-20261015`
|
|
with package ID `trial-entity-resolution-schema-BUG-000020-20261015`:
|
|
|
|
```powershell
|
|
.\.venv\Scripts\python.exe -m viano_bug_analysis.trial run --workspace service-data/trials/entity-resolution-schema-20261015 --llm-config config/llm.timeout300.local.toml --project-config config/project.local.toml --case-id BUG-000020 --package-id trial-entity-resolution-schema-BUG-000020-20261015 --stop-after interpretation --max-dispatches 6
|
|
```
|
|
|
|
## Interpretation citation policy v2
|
|
|
|
New resolution input artifacts additionally freeze
|
|
`interpretation_protocol_version = 2`. Their context uses `linked_capture_v2`
|
|
citation markers and `case_interpretation_resolved_v2`. Existing artifacts keep
|
|
the original marker-free context, prompt, validation and feedback behavior.
|
|
|
|
The selected `entity_id` must still be present in `relevant_entities`. Its
|
|
`source_ids` must include an original mention and either the entity alias or an
|
|
inline `entity_name_lookup` fact backed by the **same capture** as that selected
|
|
entity. A matching DB fact is sufficient identity evidence; a redundant N alias
|
|
is not required. An unrelated fact cannot satisfy this requirement. Missing
|
|
entities and missing provenance receive distinct error codes.
|
|
|
|
Correction feedback for the new prompt lists missing entities, the exact field,
|
|
and permitted original-mention/capture aliases. It reports these even alongside
|
|
source-kind errors, rather than waiting for multiple correction rounds to expose
|
|
each issue. Tester observations and technical facts remain strictly separated.
|
|
The prompt also distinguishes a scoped broad/localized lookup ambiguity from
|
|
an independently selected, specifically named subject; the former does not by
|
|
itself invalidate the latter.
|
|
|
|
The second real Gemini reply from the schema trial is an offline regression
|
|
fixture. It is accepted with its original response and citations under the new
|
|
policy; the old rule continues to reproduce the recorded historical rejection.
|
|
This does not overwrite or accept artifacts in the exhausted REVIEW case.
|
|
|
|
## Grouped identities and descriptive behavior (interpretation protocol v3)
|
|
|
|
Fresh resolution inputs freeze `interpretation_protocol_version = 3`. Historical
|
|
v1/v2 inputs keep their original contexts, aliases, validation and prompts.
|
|
The new final prompt is `case_interpretation_resolved_v3`; the two lookup prompts
|
|
remain protocol v2. No additional model calls are introduced.
|
|
|
|
Selected entities are grouped by exact `(type, id)` within the configured project
|
|
and lookup snapshot. A known canonical entity keeps its alias and original
|
|
provenance. Each group keeps every selected name, optional locale, role, reason,
|
|
mention citation and exact captured row in `lookup_associations`. Technical lookup
|
|
facts and raw captures are not removed. Different IDs or entity types are never
|
|
merged merely because names are similar. Any matching group capture can establish
|
|
identity in the final interpretation's citations.
|
|
|
|
The new response contract `case-interpretation-quality-output-v1.schema.json`
|
|
adds two explicitly labeled fields:
|
|
|
|
- `described_behavior`: statements derived from readable `project_description`
|
|
facts, with `basis = local_description` and
|
|
`verification = not_independently_verified`.
|
|
- `provisional_behavior_gap`: optional observation/description comparison, with
|
|
`basis = local_description`, `status = provisional`, and citations to both an
|
|
observation and a description actually included above.
|
|
|
|
These are separate from `expected_behavior`: that field continues to require
|
|
formal expectation sources, whose provenance distinguishes tester declarations
|
|
from external research. The original completeness/provenance fields and Q0001
|
|
are not promoted by a local description. `behavior_gap` and
|
|
`first_known_divergence` still require formal expectation sources. Descriptions
|
|
of prerequisites or a capture flow do not prove spawn behavior, correct runtime
|
|
implementation, a defect, or an independently verified specification.
|
|
|
|
The artifact schema binds the new response shape to the new prompt version.
|
|
Markdown summaries show the descriptive flow and provisional gap with explicit
|
|
verification labels. Hypothesis contexts retain these fields and their provenance
|
|
as separate interpretation source kinds. Empty descriptions and a null provisional
|
|
gap are valid when no applicable description or comparison is supported.
|
|
|
|
Tests cover grouping/locale/provenance retention, canonical aliases, distinct
|
|
IDs/types, valid alternative capture citations, a complete multi-query pipeline,
|
|
formal/local source separation, two-sided provisional gaps, downstream context
|
|
propagation, summaries, actual DB integration with fixture model replies, and
|
|
historical protocol replay. No paid API call is required for these checks.
|