fix(workflow): align non-motor accident extraction and evaluations
This commit is contained in:
+64
-5
@@ -1,12 +1,38 @@
|
||||
# Workflow evaluations
|
||||
|
||||
The evaluation runner calls a live FastAPI service backed by FastGPT. For now,
|
||||
each conversation step checks only the SSE `stage_code.nextStageCode` value.
|
||||
During the run it also prints the live dialogue and saves transcripts.
|
||||
The evaluation runner calls a live FastAPI service backed by FastGPT. Each
|
||||
conversation step checks the SSE `stage_code.nextStageCode` value and optional
|
||||
form / reply expectations. During the run it prints the dialogue and saves
|
||||
transcripts. Cases target `workflow/20261021`; import that workflow into FastGPT
|
||||
before running against a service.
|
||||
|
||||
## Case format
|
||||
|
||||
Add YAML files anywhere under `evals/cases/` (e.g. `direct/`, `browser_addon/`, `no_false/` for false-interrupt / false-transfer checks):
|
||||
Add YAML files anywhere under `evals/cases/`:
|
||||
|
||||
- `collection_skip_known_info/`: skip known fields and clarify missing information.
|
||||
- `non_motor_vehicle/`: both client modes, with separate flow and extraction cases.
|
||||
`all_info`, `self_collision`, `no_other_vehicle`, and `explicit_correction`
|
||||
cover conversation flow and known facts. `electric_bicycle` verifies an
|
||||
electric bicycle hitting a fixed object and transferring to `0003` without
|
||||
asking for car counts. These cases accept the described non-motor-vehicle
|
||||
collision as having `vehicle_count: 0` from the first accident description.
|
||||
`zero_count_bundle` / `zero_count_history` check zero extraction and retention.
|
||||
Explicit correction to one car must update the count to `1`.
|
||||
Both collection and extraction prompts in `workflow/20261021` use this policy
|
||||
only for an explicitly described non-motor-vehicle single-party collision with
|
||||
a fixed object; other vehicle involvement or unclear relations do not imply zero.
|
||||
- `time_confirmation/`: mandatory full-date confirmation, retaining supplemental
|
||||
information while confirmation is pending, and confirming corrected times.
|
||||
- `no_false/`: ASR noise must not cause premature transfer.
|
||||
- `photo_button_guidance/`: button help and failure handling.
|
||||
- `photo_skip_plate_confirm/`: current photo transitions without plate confirmation.
|
||||
|
||||
Obsolete scripts requiring repeated time/count confirmations or the removed
|
||||
`2004` / `2016` photo states have been removed. Photo transition cases use a
|
||||
complete accident description, confirm its standardized time, then reach their
|
||||
target branch. Time confirmation is mandatory even when all other information
|
||||
is already known; an unchanged time is confirmed only once.
|
||||
|
||||
```yaml
|
||||
name: direct_initial_stage
|
||||
@@ -26,6 +52,29 @@ steps:
|
||||
`browser_addon`). Steps in one file share the same generated `sessionId`.
|
||||
Different cases always use different session IDs.
|
||||
|
||||
Optional `expect` fields:
|
||||
|
||||
- `form`: partial equality against the cumulative form, including value types.
|
||||
- `formMatches`: field-to-regex mapping for free-text descriptions.
|
||||
- `formAbsent`: fields that must not have been extracted yet.
|
||||
- `replyMatches` / `replyNotMatches`: required / forbidden reply regexes.
|
||||
|
||||
If any step has a form expectation, the runner sends `needFormUpdate: true`
|
||||
throughout that case, merges SSE `formUpdate` deltas across turns, and records
|
||||
both updates and form snapshots. Thus an omitted unchanged field is accepted,
|
||||
while a later incorrect overwrite fails. SSE errors also fail the step.
|
||||
Existing cases without reply expectations still require manual review of their
|
||||
commented wording requirements.
|
||||
|
||||
```yaml
|
||||
- input: 没有人员伤亡。
|
||||
expect:
|
||||
stageCode: "1002"
|
||||
form: {has_injury: false, has_non_motor_vehicle_hit: true}
|
||||
replyMatches: "(什么时候|几点|事故.*时间)"
|
||||
replyNotMatches: "(有没有|是否).*非机动车"
|
||||
```
|
||||
|
||||
## Run
|
||||
|
||||
Start the API, then execute:
|
||||
@@ -45,9 +94,19 @@ Run selected cases:
|
||||
```bash
|
||||
python evals/run.py --case direct
|
||||
python evals/run.py --case browser_addon
|
||||
python evals/run.py --case no_false
|
||||
python evals/run.py --case non_motor_vehicle
|
||||
python evals/run.py --case non_motor_vehicle_electric_bicycle
|
||||
python evals/run.py --case non_motor_vehicle_zero_count
|
||||
python evals/run.py --case time_confirmation
|
||||
python evals/run.py --case photo_skip_plate_confirm
|
||||
```
|
||||
|
||||
There are 14 non-motor-vehicle cases: 10 flow cases and 4 zero-count extraction
|
||||
cases. Both groups require `vehicle_count: 0` for the described non-motor-vehicle
|
||||
collisions, while unrelated chatter must leave accident fields absent. Vehicle
|
||||
type corrections, time confirmation, and transfer checks remain active. The
|
||||
runner still stops each case at its first failure.
|
||||
|
||||
## Dialogue and transcripts
|
||||
|
||||
While a case runs, each step prints the user input, bot reply (`text_delta`),
|
||||
|
||||
Reference in New Issue
Block a user