fix(agent): reuse collected vehicle counts and avoid repeated questions
This commit is contained in:
+23
-2
@@ -3,14 +3,27 @@
|
||||
The evaluation runner calls a live FastAPI service backed by FastGPT. Each
|
||||
conversation step checks the SSE `stage_code.nextStageCode` value and optional
|
||||
form / reply expectations. During the run it prints the dialogue and saves
|
||||
transcripts. Cases target `workflow/20261021`; import that workflow into FastGPT
|
||||
before running against a service.
|
||||
transcripts. Cases cover the current LangGraph backend and workflow snapshots;
|
||||
import `workflow/20261027` when evaluating the current prompts through FastGPT.
|
||||
The original case comments retain their historical workflow versions.
|
||||
|
||||
## Case format
|
||||
|
||||
Add YAML files anywhere under `evals/cases/`:
|
||||
|
||||
- `customer_feedback/`: replay repeated connection checks and the customer's
|
||||
complete single-car collision description in both client modes. A clearly
|
||||
described motor vehicle hitting a fixed object, without other vehicle
|
||||
involvement clues, must extract `vehicle_count: 1` and skip count questions.
|
||||
Complaint wording must not discard known fields or restart confirmation.
|
||||
`python -m evals.prompt_probes --runtime --output ...` additionally replays
|
||||
the old erroneous count question to test recovery after a user challenges it.
|
||||
- `collection_skip_known_info/`: skip known fields and clarify missing information.
|
||||
The two files retaining the historical `single_no_inferred_count` name now
|
||||
follow the user-approved boundary: reversing into a fixed object, without
|
||||
other vehicle involvement clues, extracts one car. After time confirmation,
|
||||
route to single-car photography without a separate count answer. Both cases
|
||||
check `form.vehicle_count: 1` and reject repeated count/type questions.
|
||||
- `non_motor_vehicle/`: both client modes, with separate flow and extraction cases.
|
||||
`all_info`, `self_collision`, `no_other_vehicle`, and `explicit_correction`
|
||||
cover conversation flow and known facts. `electric_bicycle` verifies an
|
||||
@@ -28,6 +41,14 @@ Add YAML files anywhere under `evals/cases/`:
|
||||
- `photo_button_guidance/`: button help and failure handling.
|
||||
- `photo_skip_plate_confirm/`: current photo transitions without plate confirmation.
|
||||
|
||||
Optional semantic probes support `--exclude-case <name-fragment>`; excluded
|
||||
names are recorded separately and are never counted as passes. For the current
|
||||
quantity-only iteration, defer the historical time-anchor boundary with:
|
||||
|
||||
```bash
|
||||
python -m evals.prompt_probes --runtime --exclude-case missing_historical_time_anchor --output evals/results/quantity-probes.json
|
||||
```
|
||||
|
||||
Obsolete scripts requiring repeated time/count confirmations or the removed
|
||||
`2004` / `2016` photo states have been removed. Photo transition cases use a
|
||||
complete accident description, confirm its standardized time, then reach their
|
||||
|
||||
Reference in New Issue
Block a user