Workflow evaluations

The evaluation runner calls a live FastAPI service backed by FastGPT. Each conversation step checks the SSE stage_code.nextStageCode value and optional form / reply expectations. During the run it prints the dialogue and saves transcripts. Cases target workflow/20261021; import that workflow into FastGPT before running against a service.

Case format

Add YAML files anywhere under evals/cases/:

collection_skip_known_info/: skip known fields and clarify missing information.
non_motor_vehicle/: both client modes, with separate flow and extraction cases. all_info, self_collision, no_other_vehicle, and explicit_correction cover conversation flow and known facts. electric_bicycle verifies an electric bicycle hitting a fixed object and transferring to 0003 without asking for car counts. These cases accept the described non-motor-vehicle collision as having vehicle_count: 0 from the first accident description. zero_count_bundle / zero_count_history check zero extraction and retention. Explicit correction to one car must update the count to 1. Both collection and extraction prompts in workflow/20261021 use this policy only for an explicitly described non-motor-vehicle single-party collision with a fixed object; other vehicle involvement or unclear relations do not imply zero.
time_confirmation/: mandatory full-date confirmation, retaining supplemental information while confirmation is pending, and confirming corrected times.
no_false/: ASR noise must not cause premature transfer.
photo_button_guidance/: button help and failure handling.
photo_skip_plate_confirm/: current photo transitions without plate confirmation.

Obsolete scripts requiring repeated time/count confirmations or the removed 2004 / 2016 photo states have been removed. Photo transition cases use a complete accident description, confirm its standardized time, then reach their target branch. Time confirmation is mandatory even when all other information is already known; an unchanged time is confirmed only once.

name: direct_initial_stage
clientMode: direct

steps:
  - input: 新对话
    expect:
      stageCode: "1001"

  - input: 继续办理
    expect:
      stageCode: "1002"

clientMode in the case file is the API request setting (direct or browser_addon). Steps in one file share the same generated sessionId. Different cases always use different session IDs.

Optional expect fields:

form: partial equality against the cumulative form, including value types.
formMatches: field-to-regex mapping for free-text descriptions.
formAbsent: fields that must not have been extracted yet.
replyMatches / replyNotMatches: required / forbidden reply regexes.

If any step has a form expectation, the runner sends needFormUpdate: true throughout that case, merges SSE formUpdate deltas across turns, and records both updates and form snapshots. Thus an omitted unchanged field is accepted, while a later incorrect overwrite fails. SSE errors also fail the step. Existing cases without reply expectations still require manual review of their commented wording requirements.

  - input: 没有人员伤亡。
    expect:
      stageCode: "1002"
      form: {has_injury: false, has_non_motor_vehicle_hit: true}
      replyMatches: "(什么时候|几点|事故.*时间)"
      replyNotMatches: "(有没有|是否).*非机动车"

Run

Start the API, then execute:

python evals/run.py

Use another API address:

python evals/run.py --base-url http://192.168.1.10:8000

Run selected cases:

python evals/run.py --case direct
python evals/run.py --case browser_addon
python evals/run.py --case non_motor_vehicle
python evals/run.py --case non_motor_vehicle_electric_bicycle
python evals/run.py --case non_motor_vehicle_zero_count
python evals/run.py --case time_confirmation
python evals/run.py --case photo_skip_plate_confirm

There are 14 non-motor-vehicle cases: 10 flow cases and 4 zero-count extraction cases. Both groups require vehicle_count: 0 for the described non-motor-vehicle collisions, while unrelated chatter must leave accident fields absent. Vehicle type corrections, time confirmation, and transfer checks remain active. The runner still stops each case at its first failure.

Dialogue and transcripts

While a case runs, each step prints the user input, bot reply (text_delta), and stage-code check result.

By default, after the run finishes, transcripts are written under evals/results/<timestamp>/:

*.md — readable conversation log
*.json — structured step results

Save to a specific directory:

python evals/run.py --output-dir evals/results/manual-run --case direct

Print live dialogue only, without writing files:

python evals/run.py --no-save --case direct

The process exits with code 1 when an expectation fails and code 2 when case configuration is invalid.