fix(agent): reuse collected vehicle counts and avoid repeated questions

This commit is contained in:
Eric Wang
2026-10-28 11:30:56 +08:00
parent a277d4c7e6
commit 20c5e39e9d
15 changed files with 7073 additions and 47 deletions
+23 -2
View File
@@ -3,14 +3,27 @@
The evaluation runner calls a live FastAPI service backed by FastGPT. Each
conversation step checks the SSE `stage_code.nextStageCode` value and optional
form / reply expectations. During the run it prints the dialogue and saves
transcripts. Cases target `workflow/20261021`; import that workflow into FastGPT
before running against a service.
transcripts. Cases cover the current LangGraph backend and workflow snapshots;
import `workflow/20261027` when evaluating the current prompts through FastGPT.
The original case comments retain their historical workflow versions.
## Case format
Add YAML files anywhere under `evals/cases/`:
- `customer_feedback/`: replay repeated connection checks and the customer's
complete single-car collision description in both client modes. A clearly
described motor vehicle hitting a fixed object, without other vehicle
involvement clues, must extract `vehicle_count: 1` and skip count questions.
Complaint wording must not discard known fields or restart confirmation.
`python -m evals.prompt_probes --runtime --output ...` additionally replays
the old erroneous count question to test recovery after a user challenges it.
- `collection_skip_known_info/`: skip known fields and clarify missing information.
The two files retaining the historical `single_no_inferred_count` name now
follow the user-approved boundary: reversing into a fixed object, without
other vehicle involvement clues, extracts one car. After time confirmation,
route to single-car photography without a separate count answer. Both cases
check `form.vehicle_count: 1` and reject repeated count/type questions.
- `non_motor_vehicle/`: both client modes, with separate flow and extraction cases.
`all_info`, `self_collision`, `no_other_vehicle`, and `explicit_correction`
cover conversation flow and known facts. `electric_bicycle` verifies an
@@ -28,6 +41,14 @@ Add YAML files anywhere under `evals/cases/`:
- `photo_button_guidance/`: button help and failure handling.
- `photo_skip_plate_confirm/`: current photo transitions without plate confirmation.
Optional semantic probes support `--exclude-case <name-fragment>`; excluded
names are recorded separately and are never counted as passes. For the current
quantity-only iteration, defer the historical time-anchor boundary with:
```bash
python -m evals.prompt_probes --runtime --exclude-case missing_historical_time_anchor --output evals/results/quantity-probes.json
```
Obsolete scripts requiring repeated time/count confirmations or the removed
`2004` / `2016` photo states have been removed. Photo transition cases use a
complete accident description, confirm its standardized time, then reach their