wx44wx/ZNJJ-api-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(agent): reuse collected vehicle counts and avoid repeated questions

Browse Source
This commit is contained in:
Eric Wang
2026-10-28 11:30:56 +08:00
parent a277d4c7e6
commit 20c5e39e9d
15 changed files with 7073 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files
evals/README.md
+23 -2
View File
@@ -3,14 +3,27 @@
The evaluation runner calls a live FastAPI service backed by FastGPT. Each
conversation step checks the SSE `stage_code.nextStageCode` value and optional
form / reply expectations. During the run it prints the dialogue and saves
transcripts. Cases target `workflow/20261021`; import that workflow into FastGPT
before running against a service.
transcripts. Cases cover the current LangGraph backend and workflow snapshots;
import `workflow/20261027` when evaluating the current prompts through FastGPT.
The original case comments retain their historical workflow versions.
## Case format
Add YAML files anywhere under `evals/cases/`:
- `customer_feedback/`: replay repeated connection checks and the customer's
complete single-car collision description in both client modes. A clearly
described motor vehicle hitting a fixed object, without other vehicle
involvement clues, must extract `vehicle_count: 1` and skip count questions.
Complaint wording must not discard known fields or restart confirmation.
`python -m evals.prompt_probes --runtime --output ...` additionally replays
the old erroneous count question to test recovery after a user challenges it.
- `collection_skip_known_info/`: skip known fields and clarify missing information.
The two files retaining the historical `single_no_inferred_count` name now
follow the user-approved boundary: reversing into a fixed object, without
other vehicle involvement clues, extracts one car. After time confirmation,
route to single-car photography without a separate count answer. Both cases
check `form.vehicle_count: 1` and reject repeated count/type questions.
- `non_motor_vehicle/`: both client modes, with separate flow and extraction cases.
`all_info`, `self_collision`, `no_other_vehicle`, and `explicit_correction`
cover conversation flow and known facts. `electric_bicycle` verifies an
@@ -28,6 +41,14 @@ Add YAML files anywhere under `evals/cases/`:
- `photo_button_guidance/`: button help and failure handling.
- `photo_skip_plate_confirm/`: current photo transitions without plate confirmation.
Optional semantic probes support `--exclude-case <name-fragment>`; excluded
names are recorded separately and are never counted as passes. For the current
quantity-only iteration, defer the historical time-anchor boundary with:
```bash
python -m evals.prompt_probes --runtime --exclude-case missing_historical_time_anchor --output evals/results/quantity-probes.json
```
Obsolete scripts requiring repeated time/count confirmations or the removed
`2004` / `2016` photo states have been removed. Photo transition cases use a
complete accident description, confirm its standardized time, then reach their
evals/cases/collection_skip_known_info/browser_addon_single_no_inferred_count.yaml
+6 -7
View File
@@ -14,19 +14,18 @@ steps:
expect:
stageCode: "browser_addon.1002"
# 先确认完整时间，再询问缺失的机动车数量；不能从撞护栏推断只有一辆车。
# 用户已确认：倒车撞固定物按单方机动车事故记1；时间确认后不再问数量。
# 文件名保留历史名称，预期按最新业务边界更新。
- input: 十分钟前我倒车撞到了护栏，没有人受伤，没有撞到非机动车或行人，我还在现场。
expect:
stageCode: "browser_addon.1002"
replyMatches: "我记录的时间是.*年.*月.*日.*点.*分.*这个时间对吗"
form: {vehicle_count: 1}
replyNotMatches: "(几辆|多少辆|什么.*车型|是否.*一辆|只有.*一辆.*吗)"
- input: 对
expect:
stageCode: "browser_addon.1002"
# 明确数量后直接单车拍照，不再要求数量确认。
- input: 只有一辆汽车。
expect:
stageCode: "browser_addon.2000"
form: {vehicle_count: 1}
replyNotMatches: "(几辆|多少辆|什么.*车型|是否.*一辆|只有.*一辆.*吗|这个时间对吗)"
evals/cases/collection_skip_known_info/direct_single_no_inferred_count.yaml
+6 -7
View File
@@ -14,19 +14,18 @@ steps:
expect:
stageCode: "1002"
# 先确认完整时间，再询问缺失的机动车数量；不能从撞护栏推断只有一辆车。
# 用户已确认：倒车撞固定物按单方机动车事故记1；时间确认后不再问数量。
# 文件名保留历史名称，预期按最新业务边界更新。
- input: 十分钟前我倒车撞到了护栏，没有人受伤，没有撞到非机动车或行人，我还在现场。
expect:
stageCode: "1002"
replyMatches: "我记录的时间是.*年.*月.*日.*点.*分.*这个时间对吗"
form: {vehicle_count: 1}
replyNotMatches: "(几辆|多少辆|什么.*车型|是否.*一辆|只有.*一辆.*吗)"
- input: 对
expect:
stageCode: "1002"
# 明确数量后直接单车拍照，不再要求数量确认。
- input: 只有一辆汽车。
expect:
stageCode: "2000"
form: {vehicle_count: 1}
replyNotMatches: "(几辆|多少辆|什么.*车型|是否.*一辆|只有.*一辆.*吗|这个时间对吗)"
evals/cases/customer_feedback/browser_addon_known_fields.yaml
+20
View File
@@ -0,0 +1,20 @@
name: browser_addon_customer_feedback_known_fields
clientMode: browser_addon
# 泛化：描述对象不同、所有字段提前提供，抱怨中有效的时间肯定仍须生效。
steps:
- input: 新对话
expect: {stageCode: "browser_addon.1001"}
- input: 继续办理
expect: {stageCode: "browser_addon.1002"}
- input: 2026年10月27日14点，我这一辆汽车倒车撞到了路灯，没有其他汽车参与，所有人都没事，也没有非机动车或行人，我还在现场。
expect:
stageCode: "browser_addon.1002"
form: {vehicle_count: 1, has_injury: false, has_non_motor_vehicle_hit: false, is_on_scene: true}
replyMatches: "我记录的时间是2026年10月27日14点00分.*这个时间对吗"
replyNotMatches: "(几辆|多少辆|有没有人员受伤|是否涉及非机动车)"
- input: 对，时间没错。我这个问题上次已经回答过了，你能看到吗？
expect:
stageCode: "browser_addon.2000"
form: {vehicle_count: 1, has_injury: false, has_non_motor_vehicle_hit: false, is_on_scene: true}
replyNotMatches: "(几辆|多少辆|这个时间对吗|没.*看到)"
evals/cases/customer_feedback/browser_addon_repeated_question.yaml
+59
View File
@@ -0,0 +1,59 @@
name: browser_addon_customer_feedback_repeated_question
clientMode: browser_addon
# 客户原始反馈：连续呼叫、单车碰撞描述、时间确认后质疑重复询问。
# 完整的自身机动车撞固定物描述应提取1，后续不再问数量。
# 呼叫时整段模板重复另由语义probe检查，本例重点验证完整采集与数量分流。
steps:
- input: 新对话
expect: {stageCode: "browser_addon.1001"}
- input: 【继续办理】
expect: {stageCode: "browser_addon.1002"}
- input: 喂喂喂，有人吗？
expect:
stageCode: "browser_addon.1002"
replyMatches: "(我在|在的|收到)"
replyNotMatches: "(能听到|听得.*清)"
formAbsent: [vehicle_count, has_injury]
- input: 喂喂喂，有没有人，而且。
expect:
stageCode: "browser_addon.1002"
replyMatches: "(我在|在的|收到)"
replyNotMatches: "(能听到|听得.*清)"
formAbsent: [vehicle_count, has_injury]
- input: 喂喂喂，有人吗？
expect:
stageCode: "browser_addon.1002"
replyNotMatches: "(能听到|听得.*清)"
- input: 我的车子是自己的小轿车，撞到了路边围栏。
expect:
stageCode: "browser_addon.1002"
replyMatches: "(受伤|人伤|伤亡)"
formMatches: {accident_cause: "(围栏|护栏)"}
form: {vehicle_count: 1}
formAbsent: [has_injury]
- input: 没有人员伤伤就车子撞了。
expect:
stageCode: "browser_addon.1002"
form: {has_injury: false}
replyMatches: "(非机动车|行人)"
- input: 没有涉及非机动车，也没有行人。
expect:
stageCode: "browser_addon.1002"
form: {has_injury: false, has_non_motor_vehicle_hit: false}
replyMatches: "(什么时候|几点|事故.*时间)"
- input: 时间是2026年10月27日十四点左右。
expect:
stageCode: "browser_addon.1002"
replyMatches: "我记录的时间是2026年10月27日14点00分.*这个时间对吗"
form: {accident_time: "2026年10月27日14点0分"}
- input: 可以。
expect:
stageCode: "browser_addon.1002"
replyMatches: "(在.*现场|现场.*吗)"
replyNotMatches: "这个时间对吗"
- input: 我现在还在事故现场。
expect:
stageCode: "browser_addon.2000"
form: {vehicle_count: 1, has_injury: false, has_non_motor_vehicle_hit: false, is_on_scene: true}
replyNotMatches: "(几辆|多少辆|这个时间对吗|其他.*车.*吗|具体的数字)"
evals/cases/customer_feedback/direct_known_fields.yaml
+20
View File
@@ -0,0 +1,20 @@
name: direct_customer_feedback_known_fields
clientMode: direct
# 泛化：描述对象不同、所有字段提前提供，抱怨中有效的时间肯定仍须生效。
steps:
- input: 新对话
expect: {stageCode: "1001"}
- input: 继续办理
expect: {stageCode: "1002"}
- input: 2026年10月27日14点，我这一辆汽车倒车撞到了路灯，没有其他汽车参与，所有人都没事，也没有非机动车或行人，我还在现场。
expect:
stageCode: "1002"
form: {vehicle_count: 1, has_injury: false, has_non_motor_vehicle_hit: false, is_on_scene: true}
replyMatches: "我记录的时间是2026年10月27日14点00分.*这个时间对吗"
replyNotMatches: "(几辆|多少辆|有没有人员受伤|是否涉及非机动车)"
- input: 对，时间没错。我这个问题上次已经回答过了，你能看到吗？
expect:
stageCode: "2000"
form: {vehicle_count: 1, has_injury: false, has_non_motor_vehicle_hit: false, is_on_scene: true}
replyNotMatches: "(几辆|多少辆|这个时间对吗|没.*看到)"
evals/cases/customer_feedback/direct_repeated_question.yaml
+59
View File
@@ -0,0 +1,59 @@
name: direct_customer_feedback_repeated_question
clientMode: direct
# 客户原始反馈：连续呼叫、单车碰撞描述、时间确认后质疑重复询问。
# 完整的自身机动车撞固定物描述应提取1，后续不再问数量。
# 呼叫时整段模板重复另由语义probe检查，本例重点验证完整采集与数量分流。
steps:
- input: 新对话
expect: {stageCode: "1001"}
- input: 【继续办理】
expect: {stageCode: "1002"}
- input: 喂喂喂，有人吗？
expect:
stageCode: "1002"
replyMatches: "(我在|在的|收到)"
replyNotMatches: "(能听到|听得.*清)"
formAbsent: [vehicle_count, has_injury]
- input: 喂喂喂，有没有人，而且。
expect:
stageCode: "1002"
replyMatches: "(我在|在的|收到)"
replyNotMatches: "(能听到|听得.*清)"
formAbsent: [vehicle_count, has_injury]
- input: 喂喂喂，有人吗？
expect:
stageCode: "1002"
replyNotMatches: "(能听到|听得.*清)"
- input: 我的车子是自己的小轿车，撞到了路边围栏。
expect:
stageCode: "1002"
replyMatches: "(受伤|人伤|伤亡)"
formMatches: {accident_cause: "(围栏|护栏)"}
form: {vehicle_count: 1}
formAbsent: [has_injury]
- input: 没有人员伤伤就车子撞了。
expect:
stageCode: "1002"
form: {has_injury: false}
replyMatches: "(非机动车|行人)"
- input: 没有涉及非机动车，也没有行人。
expect:
stageCode: "1002"
form: {has_injury: false, has_non_motor_vehicle_hit: false}
replyMatches: "(什么时候|几点|事故.*时间)"
- input: 时间是2026年10月27日十四点左右。
expect:
stageCode: "1002"
replyMatches: "我记录的时间是2026年10月27日14点00分.*这个时间对吗"
form: {accident_time: "2026年10月27日14点0分"}
- input: 可以。
expect:
stageCode: "1002"
replyMatches: "(在.*现场|现场.*吗)"
replyNotMatches: "这个时间对吗"
- input: 我现在还在事故现场。
expect:
stageCode: "2000"
form: {vehicle_count: 1, has_injury: false, has_non_motor_vehicle_hit: false, is_on_scene: true}
replyNotMatches: "(几辆|多少辆|这个时间对吗|其他.*车.*吗|具体的数字)"
evals/prompt_probes.py
+89 -9
View File
@@ -8,7 +8,7 @@ import re
from pathlib import Path
from zoneinfo import ZoneInfo
from src.agent.collect import parse_state_reply, run_collect
from src.agent.collect import _build_user_message, parse_state_reply, run_collect
from src.agent.llm import build_llm_service
from src.agent.prompts import COLLECTION, EXTRACTION, render_prompt
@@ -20,7 +20,73 @@ BASE = '十分钟前两辆汽车追尾，没有人受伤，没有非机动车或
def probes():
injury_q = [{'role': 'assistant', 'content': '<state>1002</state>请问这次事故中，有没有人员受伤呢？'}]
complaint = '我这个问题上次已经回答过了，你能看到吗？'
count_question = {'role': 'assistant', 'content': '<state>1002</state>请问有几辆汽车卷入了这次事故呢？请您告诉我一个具体的数字。'}
fixed_object = '我的小轿车撞到了路边围栏，旁边还有汽车，我不清楚是否也碰到了。没有人受伤，没有非机动车或行人，我还在现场。'
confirmed = [{'role': 'assistant', 'content': CONFIRM}, {'role': 'user', 'content': '可以。'}]
known_form = dict(accident_cause='其他：车辆碰撞', has_injury=False,
has_non_motor_vehicle_hit=False, accident_time=TIME, is_on_scene=True)
return [
dict(name='feedback_single_motor_fixed_object', user='我现在还在事故现场。', history=[
{'role':'user','content':'我的车子是自己的小轿车，撞到了路边围栏。'},
{'role':'user','content':'没有人员伤伤就车子撞了。'},
{'role':'user','content':'没有涉及非机动车，也没有行人。'}, *confirmed,
{'role':'assistant','content':'<state>1002</state>请问您现在还在事故现场吗？'}],
code='2000', count=1, not_matches='(几辆|多少辆|其他.*车.*吗|这个时间对吗)'),
dict(name='feedback_single_motor_complaint', user=complaint, history=[
{'role':'user','content':'我的小轿车撞到路边围栏，没有人受伤，没有非机动车或行人，我还在现场。'},
*confirmed, count_question], code='2000', count=1,
not_matches='(几辆|多少辆|没.*看到|其他.*车.*吗)'),
dict(name='holdout_single_motor_different_object', user='我的货车倒车碰到停车场立柱，车受损了。',
history=[], code='1002', count=1, contains='受伤'),
dict(name='holdout_only_vehicle_type_not_count', user='我开的是自己的小轿车。',
history=[], code='1002', count_absent=True),
dict(name='feedback_unknown_count_acknowledge_history', user=complaint,
history=[{'role': 'user', 'content': fixed_object}, *confirmed, count_question],
code='1002', count_absent=True,
matches='(其他|还有|几辆|多少辆)',
not_matches='(没.*(看到|收到)|未.*(提供|回答)|具体的数字)'),
dict(name='feedback_known_count_in_history', user=complaint,
history=[{'role': 'user', 'content': '只有我这一辆小轿车撞到围栏。没有人受伤，没有非机动车或行人，我还在现场。'}, *confirmed, count_question],
code='2000', count=1, not_matches='(几辆|多少辆|具体的数字|没.*看到)'),
dict(name='feedback_reverse_collision_extracts_one', user='倒车时碰到了停车场墙面，车刮坏了。',
history=[], code='1002', count=1, contains='受伤'),
dict(name='feedback_form_two_skips_count', user=complaint,
history=[*confirmed, count_question], form={**known_form, 'vehicle_count': 2},
code='2010', count=2, not_matches='(几辆|多少辆|是否.*两辆|确认.*数量|没.*看到)'),
dict(name='feedback_form_three_skips_count', user=complaint,
history=[*confirmed, count_question], form={**known_form, 'vehicle_count': 3},
code='0003', count=3, not_matches='(几辆|多少辆|确认.*数量|没.*看到)'),
dict(name='feedback_form_one_other_field_missing', user='我还在现场。',
history=[*confirmed], form={'vehicle_count': 1, 'accident_cause': '倒车',
'has_non_motor_vehicle_hit': False, 'accident_time': TIME},
code='1002', count=1, contains='受伤',
not_matches='(几辆|多少辆|什么.*车型|是否.*一辆|只有.*一辆.*吗)'),
dict(name='feedback_known_count_in_form', user=complaint,
history=[*confirmed, count_question], form={**known_form, 'vehicle_count': 1},
code='2000', not_matches='(几辆|多少辆|具体的数字|没.*看到)'),
dict(name='feedback_form_false_and_unknown_count', user=complaint,
history=[*confirmed, count_question], form=known_form,
code='1002', count_absent=True,
matches='(其他|还有|几辆|多少辆|是否.*一辆|只有.*一辆)',
not_matches='(有没有人员受伤|是否涉及非机动车|现在还在事故现场|没.*看到|具体的数字)'),
dict(name='feedback_form_time_not_confirmed', user='这些我已经说过了。',
history=[], form={**known_form, 'vehicle_count': 1},
code='1002', contains='这个时间对吗'),
dict(name='feedback_correct_form_count', user='刚才的数量说错了，是两辆汽车。',
history=[*confirmed, count_question], form={**known_form, 'vehicle_count': 1},
code='2010', count=2),
dict(name='feedback_repeated_connection_check', user='喂喂喂，有人吗？', history=[
{'role':'assistant','content':'<state>1002</state>请简单描述一下事发经过，比如车辆大概是怎么撞在一起的？'},
{'role':'user','content':'喂喂喂，有人吗？'},
{'role':'assistant','content':'<state>1002</state>您好，我在的。请简单描述一下事发经过，比如车辆大概是怎么撞在一起的？'},
{'role':'user','content':'喂喂喂，有没有人，而且。'},
{'role':'assistant','content':'<state>1002</state>您好，我在的。请您简单说一下事发经过，比如车辆大概是怎么撞在一起的？'}],
code='1002', matches='(在的|我在|收到)',
not_matches='(简单.*(描述|说一下).*经过|能听到|听得.*清)'),
dict(name='feedback_no_evidence_of_previous_answer', user=complaint,
history=[count_question], code='1002', count_absent=True,
not_matches='(已记录.*一辆|已确认.*一辆|没.*看到|具体的数字)'),
dict(name='partial_symptom_negation', user='没有流血。', history=injury_q,
code='1002', injury=None, contains='受伤'),
dict(name='negation_with_positive_injury', user='没有流血，但胳膊受伤了。', history=injury_q,
@@ -33,9 +99,10 @@ def probes():
{'role':'user','content':'两辆汽车追尾，人员都没事。'},
{'role':'assistant','content':'<state>1002</state>请问事故中是否涉及非机动车或行人呢？'}],
code='1002', non_motor=None, contains='非机动车'),
dict(name='unknown_count_after_time', user='对', history=[
dict(name='single_reverse_count_after_time', user='对', history=[
{'role':'user','content':'十分钟前我倒车撞到护栏，没有人受伤，没有非机动车或行人，我还在现场。'},
{'role':'assistant','content':CONFIRM}], code='1002', contains='几辆', count_absent=True),
{'role':'assistant','content':CONFIRM}], code='2000', count=1,
not_matches='(几辆|多少辆|什么.*车型|是否.*一辆|只有.*一辆.*吗|这个时间对吗)'),
dict(name='known_count_after_time', user='对', history=[
{'role':'user','content':BASE}, {'role':'assistant','content':CONFIRM}], code='2010'),
dict(name='time_changed_requires_confirmation', user='不对，是今天十一点四十分。', history=[
@@ -70,22 +137,28 @@ async def main(args):
checks = []
evidence = {}
try:
state = {'stage_code': '1002', 'input_text': case['user'],
'dialogue': case['history'], 'accident_fields': case.get('form', {})}
if args.runtime:
result = await run_collect({'stage_code':'1002', 'input_text':case['user'],
'dialogue':case['history']}, llm, now=NOW)
result = await run_collect(state, llm, now=NOW)
raw = result['response_text']
fields = result['accident_fields']
else:
raw = await llm.complete(render_prompt(COLLECTION, {}, now=NOW), case['user'], history=case['history'])
user_message = _build_user_message(state, now=NOW)
raw = await llm.complete(render_prompt(COLLECTION, state, now=NOW), user_message, history=case['history'])
code, body = parse_state_reply(raw)
evidence['reply'] = raw
checks.append(dict(text='expected state '+case['code'], passed=code==case['code'], evidence=raw))
if 'contains' in case:
checks.append(dict(text='asks/mentions '+case['contains'], passed=case['contains'] in body, evidence=body))
for key, negate in [('matches', False), ('not_matches', True)]:
if key in case:
matched = bool(re.search(case[key], body))
checks.append(dict(text=key + ' ' + case[key], passed=matched != negate, evidence=body))
if args.runtime:
raw_fields = json.dumps(fields,ensure_ascii=False)
else:
raw_fields = await llm.complete(render_prompt(EXTRACTION, {}, now=NOW), case['user'], history=case['history'])
raw_fields = await llm.complete(render_prompt(EXTRACTION, state, now=NOW), user_message, history=case['history'])
fields = json.loads(raw_fields)
evidence['fields'] = fields
for probe_key, field in [('injury','has_injury'),('non_motor','has_non_motor_vehicle_hit')]:
@@ -93,6 +166,8 @@ async def main(args):
expected = case[probe_key]
ok = field not in fields if expected is None else fields.get(field) is expected
checks.append(dict(text=f'{field} expected {expected}', passed=ok, evidence=raw_fields))
if 'count' in case:
checks.append(dict(text='motor vehicle total ' + str(case['count']), passed=fields.get('vehicle_count') == case['count'], evidence=raw_fields))
if case.get('count_absent'):
checks.append(dict(text='no inferred motor vehicle total', passed='vehicle_count' not in fields, evidence=raw_fields))
if case.get('time_absent'):
@@ -100,8 +175,10 @@ async def main(args):
checks.append(dict(text='dialogue also avoids invented historical timestamp', passed=not re.search(r'\d{4}年.*点.*分',body), evidence=body))
except Exception as exc:
checks.append(dict(text='provider and output usable',passed=False,evidence=type(exc).__name__+': '+str(exc)))
return dict(name=case['name'], user=case['user'], history=case['history'], passed=all(c['passed'] for c in checks), checks=checks, **evidence)
selected = [c for c in probes() if not args.case or any(value in c['name'] for value in args.case)]
return dict(name=case['name'], user=case['user'], history=case['history'], form=case.get('form', {}), passed=all(c['passed'] for c in checks), checks=checks, **evidence)
selected = [c for c in probes()
if (not args.case or any(value in c['name'] for value in args.case))
and not any(value in c['name'] for value in args.exclude_case)]
if not selected:
raise SystemExit('No matching probes')
results = await asyncio.gather(*(run(c) for c in selected))
@@ -109,6 +186,8 @@ async def main(args):
from src.agent.prompts import load_prompt
profile = Config.llm_profile(Config.LLM_PROVIDER) or {}
output = dict(model=profile.get('model'), temperature=profile.get('temperature'),
probe_hash=sha256(json.dumps(selected, ensure_ascii=False, sort_keys=True).encode('utf-8')).hexdigest(),
excluded_cases=[c['name'] for c in probes() if any(value in c['name'] for value in args.exclude_case)],
execution_mode='runtime' if args.runtime else 'raw_prompt',
prompt_hashes={name:sha256(load_prompt(name).encode('utf-8')).hexdigest() for name in (COLLECTION,EXTRACTION)},
passed=sum(r['passed'] for r in results), total=len(results), results=results)
@@ -122,5 +201,6 @@ if __name__ == '__main__':
parser.add_argument('--runtime', action='store_true', help='Exercise production collection, including provenance validation')
parser.add_argument('--output', type=Path, required=True)
parser.add_argument('--case', action='append', default=[], help='Run only probe names containing this value')
parser.add_argument('--exclude-case', action='append', default=[], help='Defer probes whose names contain this value; recorded separately from passes')
parser.add_argument('--prompt-dir', type=Path, help='Optional snapshot containing collection.md and extraction.md')
asyncio.run(main(parser.parse_args()))