wx44wx/ZNJJ-api-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(evals): add shared case groups and selectable test suites

Browse Source
This commit is contained in:
Eric Wang
2026-10-28 13:51:08 +08:00
parent 3317d652db
commit ddcfe5bd05
7 changed files with 511 additions and 154 deletions
Download Patch File Download Diff File Expand all files Collapse all files
evals/README.md
+115 -130
View File
@@ -1,161 +1,146 @@
# Workflow evaluations
# 评估案例与分组运行
The evaluation runner calls a live FastAPI service backed by FastGPT. Each
conversation step checks the SSE `stage_code.nextStageCode` value and optional
form / reply expectations. During the run it prints the dialogue and saves
transcripts. Cases cover the current LangGraph backend and workflow snapshots;
import `workflow/20261027` when evaluating the current prompts through FastGPT.
The original case comments retain their historical workflow versions.
`evals/cases/` 当前保留52个完整对话案例、8个目录分组。每个文件独立创建会话，检查阶段码，以及可选的表单、回复断言。
## Case format
- `python -m evals.run`：通过HTTP调用已经启动的API服务，可测试FastGPT或LangGraph后端。
- `python -m evals.prompt_compare`：在进程内启动生产文本路由和后端生命周期，使用配置中的真实LLM测试本地LangGraph，不初始化无关语音模型。
Add YAML files anywhere under `evals/cases/`:
两个入口使用相同的案例目录、筛选参数及断言。所有Python命令使用Conda `py3.11`。
- `complex_vehicle/`: three-car collisions must immediately return `0003`
with the approved handoff wording in both client modes. Assert the first
explicit three-car turn transfers, extracts count/cause, leaves unknown
injury/time/scene fields absent, and asks no further collection or photo questions.
Semantic probes additionally cover four cars, a known count in an otherwise
empty form, new corrections, uncertain counts, unrelated vehicles, and lower bounds.
- `customer_feedback/`: replay repeated connection checks and the customer's
complete single-car collision description in both client modes. A clearly
described motor vehicle hitting a fixed object, without other vehicle
involvement clues, must extract `vehicle_count: 1` and skip count questions.
Complaint wording must not discard known fields or restart confirmation.
`python -m evals.prompt_probes --runtime --output ...` additionally replays
the old erroneous count question to test recovery after a user challenges it.
- `collection_skip_known_info/`: skip known fields and clarify missing information.
The two files retaining the historical `single_no_inferred_count` name now
follow the user-approved boundary: reversing into a fixed object, without
other vehicle involvement clues, extracts one car. After time confirmation,
route to single-car photography without a separate count answer. Both cases
check `form.vehicle_count: 1` and reject repeated count/type questions.
- `non_motor_vehicle/`: both client modes, with separate flow and extraction cases.
`all_info`, `self_collision`, `no_other_vehicle`, and `explicit_correction`
cover conversation flow and known facts. `electric_bicycle` verifies an
electric bicycle hitting a fixed object and transferring to `0003` without
asking for car counts. These cases accept the described non-motor-vehicle
collision as having `vehicle_count: 0` from the first accident description.
`zero_count_bundle` / `zero_count_history` check zero extraction and retention.
Explicit correction to one car must update the count to `1`.
Both collection and extraction prompts in `workflow/20261021` use this policy
only for an explicitly described non-motor-vehicle single-party collision with
a fixed object; other vehicle involvement or unclear relations do not imply zero.
- `time_confirmation/`: mandatory full-date confirmation, retaining supplemental
information while confirmation is pending, and confirming corrected times.
- `no_false/`: ASR noise must not cause premature transfer.
- `photo_button_guidance/`: button help and failure handling.
- `photo_skip_plate_confirm/`: current photo transitions without plate confirmation.
## 先预览，再运行
Optional semantic probes support `--exclude-case <name-fragment>`; excluded
names are recorded separately and are never counted as passes. For the current
quantity-only iteration, defer the historical time-anchor boundary with:
以下命令只读取案例，不启动应用、不调用模型、不创建结果目录，也不需要配置真实模型：
```bash
python -m evals.prompt_probes --runtime --exclude-case missing_historical_time_anchor --output evals/results/quantity-probes.json
conda run -n py3.11 python -m evals.run --list-groups
conda run -n py3.11 python -m evals.run --list-suites
conda run -n py3.11 python -m evals.run --suite smoke --mode direct --list
conda run -n py3.11 python -m evals.prompt_compare --group complex_vehicle --list
```
Obsolete scripts requiring repeated time/count confirmations or the removed
`2004` / `2016` photo states have been removed. Photo transition cases use a
complete accident description, confirm its standardized time, then reach their
target branch. Time confirmation is mandatory even when all other information
is already known; an unchanged time is confirmed only once.
`--list`列出案例名、客户端模式和相对路径；列表末尾显示本次选择数量及目录中的总数量。`--list-groups`和`--list-suites`的计数也会应用当前筛选条件。
## 目录分组与测试集
`--group`精确选择现有目录，不靠名称子串猜测分组：
| 分组 | 数量 | 主要覆盖 |
| --- | ---: | --- |
| `collection_skip_known_info` | 12 | 已知信息跳过、缺项补充、人伤不确定、单方数量推断 |
| `complex_vehicle` | 2 | 首次明确三车事故即转人工，未知字段不补造 |
| `customer_feedback` | 4 | 客户原始反馈、连续呼叫、已知信息不重复问 |
| `non_motor_vehicle` | 14 | 非机动车参与、零机动车提取、保留已知事实、类型更正 |
| `no_false` | 2 | ASR噪声不误转人工 |
| `time_confirmation` | 2 | 独立确认完整时间、补充信息保留、更正后重新确认 |
| `photo_button_guidance` | 12 | 单双车按钮指引、失败处理、不重复说明 |
| `photo_skip_plate_confirm` | 4 | 当前拍照步骤转换，不额外确认车牌 |
`--suite`选择跨目录测试集，定义集中在[suites.yaml](suites.yaml)，只引用案例路径，不复制输入或断言：
| 测试集 | 两种模式合计 | 适用场景 |
| --- | ---: | --- |
| `smoke` | 12 | 修改后的快速回归；配合`--mode direct`仅6个案例 |
| `collection` | 36 | 采集提示词、表单保持、时间与事故分流 |
| `vehicle_count` | 22 | 数量推断、零机动车、类型更正、三车和客户反馈 |
| `safety` | 20 | 复杂事故、非机动车、ASR噪声、不确定人伤 |
| `photo` | 16 | 拍照节点和按钮交互 |
| `feedback` | 6 | 客户反馈，包括三车立即转人工 |
| `full` | 52 | 全量回归；不提供筛选条件也执行全量 |
详细场景索引及保留历史文件名的原因见[cases/README.md](cases/README.md)。
## 局部测试示例
先启动待测试API，再运行远程HTTP评估：
```bash
# 只测试三车分流，先预览再执行
conda run -n py3.11 python -m evals.run --group complex_vehicle --mode direct --list
conda run -n py3.11 python -m evals.run --group complex_vehicle --mode direct
# 数量相关案例，仅direct客户端
conda run -n py3.11 python -m evals.run --suite vehicle_count --mode direct
# 同时选两个目录
conda run -n py3.11 python -m evals.run --group time_confirmation --group complex_vehicle
# 拍照测试集内只选双车，排除按钮失败场景
conda run -n py3.11 python -m evals.run --suite photo --case dual --exclude-case failure
# 指定服务及结果目录
conda run -n py3.11 python -m evals.run --suite smoke --base-url http://127.0.0.1:8000 --output-dir evals/results/smoke
```
本地LangGraph评估使用完全相同的筛选方式：
```bash
conda run -n py3.11 python -m evals.prompt_compare --suite smoke --mode direct --output-dir evals/results/local-smoke
conda run -n py3.11 python -m evals.prompt_compare --suite feedback --output-dir evals/results/local-feedback
conda run -n py3.11 python -m evals.prompt_compare --suite full --workers 4 --output-dir evals/results/local-full
```
执行本地评估需要配置`AGENT_BACKEND=langgraph`和真实的`LLM_PROVIDER`；建议用`AGENT_STORAGE=memory`隔离评估会话。`--output-dir`在实际执行时必填，预览时不必填。`metadata.json`记录筛选条件、选中的案例名、所有案例及提示词哈希，便于核对本轮覆盖范围。
远程入口继续支持`python evals/run.py`、`EVAL_BASE_URL`、`--timeout`、`--insecure`、`--no-save`；本地入口继续支持`--prompt-dir`提示词快照和`--workers`并发度。两个入口均支持`--cases-dir`指定其他案例目录。
## 筛选与去重规则
- 同类参数内部取并集：重复`--group`、`--suite`或`--case`表示任意一个匹配。
- 不同参数之间取交集：例如`--suite photo --mode direct --case dual`只选拍照集中的direct双车案例。
- `--case`保持原有的案例名子串匹配；`--exclude-case`最后按名称子串排除。
- 重叠测试集中的同一个文件只执行一次，不因重复指定测试集而复制会话。
- 未知分组、未知测试集或空选择返回配置错误，避免把“没有跑任何案例”当作成功。
- 案例名必须唯一，避免输出文件相互覆盖。
此次整理未删除、移动或改写任何原案例。核对结果：同一客户端模式内，没有输入和断言完全相同的重复文件，也没有完整输入序列相同的文件。direct和browser_addon的相似案例验证不同请求模式与阶段码，保留为独立覆盖；只想测一种模式时使用`--mode`减少执行量。共同的欢迎和时间确认轮次是对话上下文的一部分，不单独抽掉。
## 新增案例与断言
把YAML放入现有目录，或新建业务目录；目录自动成为`--group`。新案例会自动纳入全量执行；按需要更新`suites.yaml`中的跨目录路径规则。套件定义放在cases目录外，避免被当成对话案例读取。
```yaml
name: direct_initial_stage
name: direct_example
clientMode: direct
steps:
- input: 新对话
expect:
stageCode: "1001"
- input: 继续办理
expect:
stageCode: "1002"
```
`clientMode` in the case file is the API request setting (`direct` or
`browser_addon`). Steps in one file share the same generated `sessionId`.
Different cases always use different session IDs.
`clientMode`为`direct`或`browser_addon`，后者的阶段断言需要`browser_addon.`前缀。同一文件的各轮共享一个新生成的sessionId，不同文件使用不同sessionId。每个案例在首个失败轮次停止，并尝试清理会话。
Optional `expect` fields:
可选断言：
- `form`: partial equality against the cumulative form, including value types.
- `formMatches`: field-to-regex mapping for free-text descriptions.
- `formAbsent`: fields that must not have been extracted yet.
- `replyMatches` / `replyNotMatches`: required / forbidden reply regexes.
| 字段 | 含义 |
| --- | --- |
| `form` | 累计表单的部分精确相等，包含类型检查 |
| `formMatches` | 自由文本字段的正则检查 |
| `formAbsent` | 尚不得提取的字段 |
| `replyMatches` | 回复必须匹配的正则 |
| `replyNotMatches` | 回复禁止匹配的正则 |
If any step has a form expectation, the runner sends `needFormUpdate: true`
throughout that case, merges SSE `formUpdate` deltas across turns, and records
both updates and form snapshots. Thus an omitted unchanged field is accepted,
while a later incorrect overwrite fails. SSE errors also fail the step.
Existing cases without reply expectations still require manual review of their
commented wording requirements.
一个案例包含任意表单断言时，执行器全程请求`needFormUpdate: true`，跨轮合并SSE表单增量。因此未重复返回不变字段可以接受，后续错误覆盖会失败。SSE错误也使本轮失败。仅有注释、没有回复断言的旧案例仍需要查看实际对话。
```yaml
- input: 没有人员伤亡。
expect:
stageCode: "1002"
form: {has_injury: false, has_non_motor_vehicle_hit: true}
replyMatches: "(什么时候|几点|事故.*时间)"
replyNotMatches: "(有没有|是否).*非机动车"
```
当前边界：完整倒车撞固定物且无其他车辆参与线索按1辆处理；已知至少3辆机动车参与立即转人工；单/双车进入拍照前需要独立确认时间。非机动车参与的既有分支保留必要采集及时间确认。通过FastGPT评估当前提示词时，应先导入本仓库`workflow/20261027`快照；案例中的历史版本注释保留供追溯。
## Run
## 结果与补充语义检查
Start the API, then execute:
运行时打印每轮输入、回复和断言结果。远程入口默认保存到`evals/results/<timestamp>/`；本地入口保存到指定目录：
- `*.md`：可读对话记录。
- `*.json`：每轮阶段码、回复、表单增量、累计表单和断言。
- 本地入口另保存`metadata.json`、`summary.json`及运行日志。
两个入口实际执行时，任一案例失败返回1，配置错误返回2，通过返回0。`--list`等预览成功返回0。
`prompt_probes.py`用于局部状态快照和历史回放，独立于cases；`time_boundary.py`检查历史时间来源边界。它们不读取`suites.yaml`。示例：
```bash
python evals/run.py
conda run -n py3.11 python -m evals.prompt_probes --runtime --case complex_ --output evals/results/complex-probes.json
conda run -n py3.11 python -m evals.prompt_probes --runtime --exclude-case missing_historical_time_anchor --output evals/results/probes.json
```
Use another API address:
```bash
python evals/run.py --base-url http://192.168.1.10:8000
```
Run selected cases:
```bash
python evals/run.py --case direct
python evals/run.py --case browser_addon
python evals/run.py --case non_motor_vehicle
python evals/run.py --case non_motor_vehicle_electric_bicycle
python evals/run.py --case non_motor_vehicle_zero_count
python evals/run.py --case time_confirmation
python evals/run.py --case photo_skip_plate_confirm
```
There are 14 non-motor-vehicle cases: 10 flow cases and 4 zero-count extraction
cases. Both groups require `vehicle_count: 0` for the described non-motor-vehicle
collisions, while unrelated chatter must leave accident fields absent. Vehicle
type corrections, time confirmation, and transfer checks remain active. The
runner still stops each case at its first failure.
## Dialogue and transcripts
While a case runs, each step prints the user input, bot reply (`text_delta`),
and stage-code check result.
By default, after the run finishes, transcripts are written under
`evals/results/<timestamp>/`:
- `*.md` — readable conversation log
- `*.json` — structured step results
Save to a specific directory:
```bash
python evals/run.py --output-dir evals/results/manual-run --case direct
```
Print live dialogue only, without writing files:
```bash
python evals/run.py --no-save --case direct
```
The process exits with code `1` when an expectation fails and code `2` when
case configuration is invalid.
`missing_historical_time_anchor`是此前用户要求暂缓的历史时间基准检查；仅显式排除时跳过，排除项单独记录，不能计为通过。本次整理没有改变这个边界。