# Workflow evaluations

The evaluation runner calls a live FastAPI service backed by FastGPT. Each
conversation step checks the SSE `stage_code.nextStageCode` value and optional
form / reply expectations. During the run it prints the dialogue and saves
transcripts. Cases cover the current LangGraph backend and workflow snapshots;
import `workflow/20261027` when evaluating the current prompts through FastGPT.
The original case comments retain their historical workflow versions.

## Case format

Add YAML files anywhere under `evals/cases/`:

- `complex_vehicle/`: three-car collisions must immediately return `0003`
  with the approved handoff wording in both client modes. Assert the first
  explicit three-car turn transfers, extracts count/cause, leaves unknown
  injury/time/scene fields absent, and asks no further collection or photo questions.
  Semantic probes additionally cover four cars, a known count in an otherwise
  empty form, new corrections, uncertain counts, unrelated vehicles, and lower bounds.
- `customer_feedback/`: replay repeated connection checks and the customer's
  complete single-car collision description in both client modes. A clearly
  described motor vehicle hitting a fixed object, without other vehicle
  involvement clues, must extract `vehicle_count: 1` and skip count questions.
  Complaint wording must not discard known fields or restart confirmation.
  `python -m evals.prompt_probes --runtime --output ...` additionally replays
  the old erroneous count question to test recovery after a user challenges it.
- `collection_skip_known_info/`: skip known fields and clarify missing information.
  The two files retaining the historical `single_no_inferred_count` name now
  follow the user-approved boundary: reversing into a fixed object, without
  other vehicle involvement clues, extracts one car. After time confirmation,
  route to single-car photography without a separate count answer. Both cases
  check `form.vehicle_count: 1` and reject repeated count/type questions.
- `non_motor_vehicle/`: both client modes, with separate flow and extraction cases.
  `all_info`, `self_collision`, `no_other_vehicle`, and `explicit_correction`
  cover conversation flow and known facts. `electric_bicycle` verifies an
  electric bicycle hitting a fixed object and transferring to `0003` without
  asking for car counts. These cases accept the described non-motor-vehicle
  collision as having `vehicle_count: 0` from the first accident description.
  `zero_count_bundle` / `zero_count_history` check zero extraction and retention.
  Explicit correction to one car must update the count to `1`.
  Both collection and extraction prompts in `workflow/20261021` use this policy
  only for an explicitly described non-motor-vehicle single-party collision with
  a fixed object; other vehicle involvement or unclear relations do not imply zero.
- `time_confirmation/`: mandatory full-date confirmation, retaining supplemental
  information while confirmation is pending, and confirming corrected times.
- `no_false/`: ASR noise must not cause premature transfer.
- `photo_button_guidance/`: button help and failure handling.
- `photo_skip_plate_confirm/`: current photo transitions without plate confirmation.

Optional semantic probes support `--exclude-case <name-fragment>`; excluded
names are recorded separately and are never counted as passes. For the current
quantity-only iteration, defer the historical time-anchor boundary with:

```bash
python -m evals.prompt_probes --runtime --exclude-case missing_historical_time_anchor --output evals/results/quantity-probes.json
```

Obsolete scripts requiring repeated time/count confirmations or the removed
`2004` / `2016` photo states have been removed. Photo transition cases use a
complete accident description, confirm its standardized time, then reach their
target branch. Time confirmation is mandatory even when all other information
is already known; an unchanged time is confirmed only once.

```yaml
name: direct_initial_stage
clientMode: direct

steps:
  - input: 新对话
    expect:
      stageCode: "1001"

  - input: 继续办理
    expect:
      stageCode: "1002"
```

`clientMode` in the case file is the API request setting (`direct` or
`browser_addon`). Steps in one file share the same generated `sessionId`.
Different cases always use different session IDs.

Optional `expect` fields:

- `form`: partial equality against the cumulative form, including value types.
- `formMatches`: field-to-regex mapping for free-text descriptions.
- `formAbsent`: fields that must not have been extracted yet.
- `replyMatches` / `replyNotMatches`: required / forbidden reply regexes.

If any step has a form expectation, the runner sends `needFormUpdate: true`
throughout that case, merges SSE `formUpdate` deltas across turns, and records
both updates and form snapshots. Thus an omitted unchanged field is accepted,
while a later incorrect overwrite fails. SSE errors also fail the step.
Existing cases without reply expectations still require manual review of their
commented wording requirements.

```yaml
  - input: 没有人员伤亡。
    expect:
      stageCode: "1002"
      form: {has_injury: false, has_non_motor_vehicle_hit: true}
      replyMatches: "(什么时候|几点|事故.*时间)"
      replyNotMatches: "(有没有|是否).*非机动车"
```

## Run

Start the API, then execute:

```bash
python evals/run.py
```

Use another API address:

```bash
python evals/run.py --base-url http://192.168.1.10:8000
```

Run selected cases:

```bash
python evals/run.py --case direct
python evals/run.py --case browser_addon
python evals/run.py --case non_motor_vehicle
python evals/run.py --case non_motor_vehicle_electric_bicycle
python evals/run.py --case non_motor_vehicle_zero_count
python evals/run.py --case time_confirmation
python evals/run.py --case photo_skip_plate_confirm
```

There are 14 non-motor-vehicle cases: 10 flow cases and 4 zero-count extraction
cases. Both groups require `vehicle_count: 0` for the described non-motor-vehicle
collisions, while unrelated chatter must leave accident fields absent. Vehicle
type corrections, time confirmation, and transfer checks remain active. The
runner still stops each case at its first failure.

## Dialogue and transcripts

While a case runs, each step prints the user input, bot reply (`text_delta`),
and stage-code check result.

By default, after the run finishes, transcripts are written under
`evals/results/<timestamp>/`:

- `*.md` — readable conversation log
- `*.json` — structured step results

Save to a specific directory:

```bash
python evals/run.py --output-dir evals/results/manual-run --case direct
```

Print live dialogue only, without writing files:

```bash
python evals/run.py --no-save --case direct
```

The process exits with code `1` when an expectation fails and code `2` when
case configuration is invalid.