为 EOT Studio 增加可展开的模型分数时间轴
This commit is contained in:
Xin Wang
@@ -24,6 +24,27 @@ npm run dev
|
||||
|
||||
浏览器草稿自动保留，刷新后可恢复。网页保存不覆盖原 XLSX、初稿和录音。初稿转换方法、标签映射与限制见 [转换说明](../EOT标注/韩泰初稿/转换说明.md)，原行号及原结论保留在初稿目录的 `source_rows.json` 中。原表没有时间信息，录音中未收录的开场与其他发言需人工补充。
|
||||
|
||||
## 模型 EOT 分数
|
||||
|
||||
点击时间轴上方的“模型 EOT 分数”展开对比，默认收起。MaAI（紫色）和 Smart Turn（青色）共用录音的时间刻度、缩放、播放头及判断点参考线。点击或拖动曲线区域定位播放，悬停查看当时已产生的分数；点击空心评分点定位到原标注时刻。顶部显示播放位置最近的有效评分，暂无评分时显示“—”，不将缺失数据当作 0。
|
||||
|
||||
- 实线阶梯曲线来自对六通完整录音的真实推理，不是用44个点插值。MaAI每100ms输入一块音频，因流式重采样缓冲，部分评分间隔为200ms；Smart Turn每200ms计算一次，并包含原标注的精确时刻。
|
||||
- MaAI使用双声道连续回放、20秒KV缓存，分数为 `1 - p_now[user]`，是接话候选分数，不是校准的EOT概率。Smart Turn沿用原回测的用户声道、过去ASR定位、500ms前置音频、3秒内续说及最多8秒窗口；没有额外的结束信号、VAD停止门槛或超时放行。
|
||||
- 空心点保留原44点的独立窗口回测分数。尤其MaAI的连续状态与独立窗口不同，空心点可能不落在连续曲线上。原始44点指标不会被这次时间序列覆盖。
|
||||
- 参考阈值只改变虚线位置，不修改人工标签。MaAI可选0.11、0.5、0.8，Smart Turn可选0.5、0.8。保存的分数不会随标注或转写编辑而重新计算。
|
||||
- 仅在展开时读取当前项目录音的评分，并核对音频SHA256。更换为本地新录音、音频变化或没有对应推理数据时，不显示旧曲线。切换通话会取消旧请求，避免前一段数据覆盖当前段。
|
||||
|
||||
评分数据位于各模型实验目录的 `results/score-timeline.json`。更新数据时，在两个各自的本机uv环境中运行：
|
||||
|
||||
```bash
|
||||
cd experiments/maai-ch-kyoto
|
||||
uv run --locked python export_timeline.py
|
||||
cd ../smart-turn-v3.2
|
||||
uv run --locked python export_timeline.py
|
||||
```
|
||||
|
||||
上述命令从项目根目录开始；生成脚本只读取录音及已有回测资料，不改人工标注。时间戳表示该评分输入已收到的时刻，MaAI使用流式重采样并计入其缓冲；Smart Turn先裁取过去音频再重采样，均不读取判断时刻之后的音频。
|
||||
|
||||
## CSV 格式
|
||||
|
||||
对话转写 `asr_history.csv`：
|
||||
@@ -48,6 +69,7 @@ demo,5200,我明天下午,0,high,mid_turn_pause,A
|
||||
## 本地接口
|
||||
|
||||
- `GET /api/workspace`：读取工作数据和项目录音列表。工作数据的两份 CSV 都存在时使用它们，否则载入韩泰初稿；初稿不存在时载入示例；已保存 CSV 格式错误时返回错误，不用示例覆盖。
|
||||
- `GET /api/model-scores?callId=...&recordingPath=...`：只读固定实验目录的已验证评分，按通话ID和项目录音路径精确匹配，并核对实际音频SHA256。缺少某模型时保留另一模型；未匹配或过期的评分不返回曲线。
|
||||
- `POST /api/workspace`：接受 `{ transcripts, annotations }`，校验后保存到固定工作数据目录，并返回 `savedAt`。同一进程的读写按顺序执行；文件锁防止不同进程同时操作，请求完成后释放。若进程在保存中被强制终止，请确认没有其他服务运行，检查残留 `.backup` 文件并恢复同一次保存前的两份 CSV，再移除工作数据目录的 `.workspace.lock`。
|
||||
- `GET /api/audio?path=...`：仅提供 `data` 内的 MP3、WAV、M4A、OGG、FLAC 和 WebM 录音，支持 Range 请求和 HEAD，便于跳转播放。路径相对 `data` 目录，不能通过 `..` 或符号链接访问目录之外的文件。
|
||||
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
import { promises as fs } from "node:fs";
|
||||
import { createHash } from "node:crypto";
|
||||
import path from "node:path";
|
||||
import { parseScoreExport, type ModelId, type ScoreResponse } from "@/lib/model-scores";
|
||||
|
||||
export const runtime = "nodejs";
|
||||
export const dynamic = "force-dynamic";
|
||||
const root = path.resolve(process.cwd(), "..", "..");
|
||||
const sources: { model: ModelId; directory: string; name: string }[] = [
|
||||
{ model: "maai", directory: "maai-ch-kyoto", name: "MaAI" },
|
||||
{ model: "smart_turn", directory: "smart-turn-v3.2", name: "Smart Turn" },
|
||||
];
|
||||
const json = (data: unknown, status = 200) => Response.json(data, { status, headers: { "Cache-Control": "no-store" } });
|
||||
|
||||
export async function GET(request: Request) {
|
||||
const params = new URL(request.url).searchParams;
|
||||
const callId = params.get("callId") || "";
|
||||
const recordingPath = params.get("recordingPath") || "";
|
||||
if (!callId || callId.length > 200 || !recordingPath || recordingPath.length > 1000) {
|
||||
return json({ error: "请选择已关联项目录音的对话。" }, 400);
|
||||
}
|
||||
const payload: ScoreResponse = { callId, recordingPath, series: [], messages: [] };
|
||||
try {
|
||||
for (const source of sources) {
|
||||
try {
|
||||
// Fixed paths only: no request value is used to locate an export.
|
||||
const file = path.join(root, "experiments", source.directory, "results", "score-timeline.json");
|
||||
const data = parseScoreExport(JSON.parse(await fs.readFile(file, "utf8")), source.model);
|
||||
const call = data.calls.find(item => item.call_id === callId && item.audio_path === recordingPath);
|
||||
if (call) payload.series.push({ ...call, model: data.model, name: data.name,
|
||||
score: data.score, sample_interval_ms: data.sample_interval_ms });
|
||||
else payload.messages.push(`${source.name} 暂无这段录音的评分。`);
|
||||
} catch (error) {
|
||||
if (!(error instanceof Error && "code" in error && error.code === "ENOENT")) {
|
||||
console.error(`Cannot read ${source.name} scores`, error);
|
||||
}
|
||||
payload.messages.push(`${source.name} 评分暂时不可用。`);
|
||||
}
|
||||
}
|
||||
if (payload.series.length) {
|
||||
const dataRoot = await fs.realpath(path.join(root, "data"));
|
||||
const audio = await fs.realpath(path.resolve(dataRoot, payload.series[0].audio_path));
|
||||
const relative = path.relative(dataRoot, audio);
|
||||
if (relative.startsWith(`..${path.sep}`) || relative === ".." || path.isAbsolute(relative)) {
|
||||
return json({ error: "评分对应的录音不在项目目录内。" }, 422);
|
||||
}
|
||||
const hash = createHash("sha256").update(await fs.readFile(audio)).digest("hex");
|
||||
payload.series = payload.series.filter(series => {
|
||||
if (series.audio_sha256 === hash) return true;
|
||||
payload.messages.push(`${series.name} 的评分对应旧录音，请重新生成评分。`);
|
||||
return false;
|
||||
});
|
||||
}
|
||||
return json(payload);
|
||||
} catch (error) {
|
||||
console.error("Cannot verify score recording", error);
|
||||
return json({ error: "无法核对评分对应的录音，请检查项目文件。" }, 500);
|
||||
}
|
||||
}
|
||||
@@ -357,7 +357,7 @@ export default function Workbench() {
|
||||
<main className="main-workspace"><div className="page-heading"><div><div className="breadcrumb">工作空间 <span>/</span> {callName}</div><h1>对话标注<span className="offline-badge">离线转写</span></h1><p title={callId}>{callId === "demo" ? "从一段对话开始，判断用户是否已经说完。" : callId}</p></div><div className="view-switch" aria-label="工作模式"><button className={mode === "edit" ? "active" : ""} onClick={() => setMode("edit")}><Icon name="edit" size={15} />标注</button><button className={mode === "view" ? "active" : ""} onClick={() => { setMode("view"); setTranscriptDraft(null); }}>查看</button></div></div>
|
||||
<section className="audio-card panel"><div className="panel-heading"><div><h2><Icon name="headphones" />录音与时间轴</h2><p className="audio-filename" title={recording?.name}>{recording?.name || "示例尚未关联录音，可先查看转写与标注"}</p></div><button className="button small ghost" onClick={() => audioImportRef.current?.click()}><Icon name="upload" size={15} />{recording ? "更换录音" : "关联录音"}</button></div>
|
||||
<div className="player"><div className="player-main"><button className="icon-button" aria-label="后退 3 秒" onClick={() => seek(currentMs - 3000)}><Icon name="back" /></button><button className="play-button" aria-label={playing ? "暂停录音" : "播放录音"} onClick={() => void togglePlay()} disabled={!recording}><Icon name={playing ? "pause" : "play"} size={19} /></button><button className="icon-button" aria-label="前进 3 秒" onClick={() => seek(currentMs + 3000)}><Icon name="forward" /></button><span className="time-display">{formatTime(currentMs)}<span> / {formatTime(durationMs)}</span></span></div><div className="player-secondary"><label className="speed-label">播放速度<select aria-label="播放速度" value={rate} onChange={event => setRate(Number(event.target.value))}>{[.75, 1, 1.25, 1.5, 2].map(value => <option key={value} value={value}>{value}×</option>)}</select></label>{mode === "edit" && <div className="menu-wrap"><button className="button small subtle" onClick={() => setMenu(menu === "create" ? null : "create")} disabled={loading} aria-expanded={menu === "create"} aria-haspopup="menu"><Icon name="plus" size={16} />在播放位置新增<Icon name="chevron" size={13} /></button>{menu === "create" && <div className="dropdown" role="menu" aria-label="新增项目"><button role="menuitem" onClick={addAnnotation}>EOT 判断点</button><button role="menuitem" onClick={() => addTranscriptAt("agent")}>机器人转写</button><button role="menuitem" onClick={() => addTranscriptAt("user")}>用户转写</button></div>}</div>}</div></div>
|
||||
<Timeline durationMs={durationMs} currentMs={currentMs} decisionMs={selected?.decision_ms ?? null} transcripts={callTranscripts.map(({ row, index }) => ({ ...row, index }))} annotations={callAnnotations} selectedId={selectedId} waveform={waveform} onSeek={seek} onMoveAnnotation={moveAnnotation} onSelectAnnotation={selectAnnotation} onSelectTranscript={index => {
|
||||
<Timeline callId={callId} recordingPath={recording?.url.startsWith("/api/audio?") ? recording.path : null} durationMs={durationMs} currentMs={currentMs} decisionMs={selected?.decision_ms ?? null} transcripts={callTranscripts.map(({ row, index }) => ({ ...row, index }))} annotations={callAnnotations} selectedId={selectedId} waveform={waveform} onSeek={seek} onMoveAnnotation={moveAnnotation} onSelectAnnotation={selectAnnotation} onSelectTranscript={index => {
|
||||
const row = transcripts[index]; seek(row.start_ms);
|
||||
audioRef.current?.pause(); setTranscriptDraft({ index, row: { ...row } });
|
||||
}} onResizeTranscript={resizeTranscript} editingTranscriptIndex={transcriptDraft?.index ?? null} onAddAnnotation={addAnnotationAt} readOnly={mode === "view"} />
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
"use client";
|
||||
|
||||
import { useEffect, useMemo, useState } from "react";
|
||||
import { scoreAt, scorePath, type ModelId, type ScoreResponse, type ScoreSeries } from "@/lib/model-scores";
|
||||
import type { TimelineAnnotation } from "./timeline";
|
||||
import styles from "./timeline.module.css";
|
||||
|
||||
const EMPTY: ScoreSeries[] = [];
|
||||
const COLORS: Record<ModelId, string> = { maai: "#8866b2", smart_turn: "#278d95" };
|
||||
export const SCORE_HEIGHT = 172;
|
||||
export type ScoreThresholds = Record<ModelId, number>;
|
||||
const time = (ms: number) => `${Math.floor(ms / 60000).toString().padStart(2, "0")}:${(ms % 60000 / 1000).toFixed(3).padStart(6, "0")}`;
|
||||
|
||||
export function useModelScores(callId: string, recordingPath: string | null, expanded: boolean) {
|
||||
const key = `${callId}:${recordingPath}`;
|
||||
const [retry, setRetry] = useState(0);
|
||||
const [state, setState] = useState<{ key: string; status: "loading" | "ready" | "error"; data?: ScoreResponse; error?: string } | null>(null);
|
||||
useEffect(() => {
|
||||
if (!expanded || !recordingPath) return;
|
||||
const controller = new AbortController();
|
||||
setState({ key, status: "loading" });
|
||||
void (async () => {
|
||||
try {
|
||||
const query = new URLSearchParams({ callId, recordingPath });
|
||||
const response = await fetch(`/api/model-scores?${query}`, { cache: "no-store", signal: controller.signal });
|
||||
const data = await response.json();
|
||||
if (!response.ok) throw new Error(data.error || "评分读取失败。");
|
||||
if (data.callId !== callId || data.recordingPath !== recordingPath || !Array.isArray(data.series)) throw new Error("评分与当前录音不匹配。");
|
||||
if (!controller.signal.aborted) setState({ key, status: "ready", data });
|
||||
} catch (error) {
|
||||
if (!controller.signal.aborted) setState({ key, status: "error", error: error instanceof Error ? error.message : "评分读取失败。" });
|
||||
}
|
||||
})();
|
||||
return () => controller.abort();
|
||||
}, [callId, recordingPath, expanded, key, retry]);
|
||||
const active = state?.key === key ? state : null;
|
||||
return {
|
||||
series: recordingPath && active?.status === "ready" ? active.data?.series || EMPTY : EMPTY,
|
||||
status: !recordingPath ? "empty" : active?.status || "loading",
|
||||
messages: !recordingPath ? ["请使用项目原录音查看模型评分；新关联的录音需先生成对应评分。"]
|
||||
: active?.status === "error" ? [active.error || "评分读取失败。"] : active?.data?.messages || [],
|
||||
retry: () => setRetry(value => value + 1),
|
||||
};
|
||||
}
|
||||
|
||||
export function ScoreControls({ series, currentMs, thresholds, onThreshold }: {
|
||||
series: ScoreSeries[]; currentMs: number; thresholds: ScoreThresholds;
|
||||
onThreshold: (model: ModelId, value: number) => void;
|
||||
}) {
|
||||
return <div className={styles.scoreControls}>
|
||||
{series.map(row => {
|
||||
const point = scoreAt(row.points, currentMs, row.sample_interval_ms * 2);
|
||||
return <div className={styles.modelControl} key={row.model} style={{ color: COLORS[row.model] }}>
|
||||
<span className={styles.modelSwatch} /><strong>{row.name}</strong>
|
||||
<output aria-live="off" title={point ? `评分时刻 ${time(point[0])}` : "当前位置暂无评分"}>{point ? point[1].toFixed(3) : "—"}</output>
|
||||
<label>参考线<select aria-label={`${row.name} 参考阈值`} value={thresholds[row.model]}
|
||||
onChange={event => onThreshold(row.model, Number(event.target.value))}>
|
||||
{(row.model === "maai" ? [.11, .5, .8] : [.5, .8]).map(value => <option key={value} value={value}>{value.toFixed(2)}</option>)}
|
||||
</select></label>
|
||||
</div>;
|
||||
})}
|
||||
</div>;
|
||||
}
|
||||
|
||||
export default function ModelScoreTrack({ series, width, durationMs, currentMs, thresholds, annotations, onSeek, onSelectAnnotation }: {
|
||||
series: ScoreSeries[]; width: number; durationMs: number; currentMs: number;
|
||||
thresholds: ScoreThresholds; annotations: TimelineAnnotation[];
|
||||
onSeek: (ms: number) => void; onSelectAnnotation: (id: string, seekToPoint?: boolean) => void;
|
||||
}) {
|
||||
const [hoverMs, setHoverMs] = useState<number | null>(null);
|
||||
const paths = useMemo(() => series.map(row => ({ ...row,
|
||||
path: scorePath(row.points, durationMs, width, SCORE_HEIGHT, row.sample_interval_ms) })), [series, durationMs, width]);
|
||||
const y = (p: number) => 12 + (1 - p) * (SCORE_HEIGHT - 24);
|
||||
const x = (ms: number) => ms / durationMs * width;
|
||||
const inspect = hoverMs ?? currentMs;
|
||||
const selectPoint = (ms: number) => {
|
||||
const annotation = annotations.find(row => row.decision_ms === ms);
|
||||
if (annotation) onSelectAnnotation(annotation.id, false);
|
||||
onSeek(ms);
|
||||
};
|
||||
return <div className={styles.scoreTrack} onPointerLeave={() => setHoverMs(null)}>
|
||||
<svg width={width} height={SCORE_HEIGHT} viewBox={`0 0 ${width} ${SCORE_HEIGHT}`}
|
||||
role="group" aria-label="MaAI 与 Smart Turn 的 EOT 分数，纵轴 0 到 1，横轴与录音时间一致"
|
||||
onPointerMove={event => {
|
||||
const bounds = event.currentTarget.getBoundingClientRect();
|
||||
setHoverMs(Math.max(0, Math.min(durationMs, (event.clientX - bounds.left) / bounds.width * durationMs)));
|
||||
}}>
|
||||
{[0, .25, .5, .75, 1].map(p => <line key={p} x1={0} x2={width} y1={y(p)} y2={y(p)} stroke="#e8e9e4" strokeDasharray={p === 0 || p === 1 ? undefined : "2 4"} />)}
|
||||
{annotations.map(row => <line key={row.id} x1={x(row.decision_ms)} x2={x(row.decision_ms)} y1={0} y2={SCORE_HEIGHT}
|
||||
stroke={row.can_reply_now === "0" ? "#c68c6b" : "#96a781"} strokeDasharray="2 5" opacity=".4" />)}
|
||||
{paths.map(row => <g key={row.model}>
|
||||
<line x1={0} x2={width} y1={y(thresholds[row.model])} y2={y(thresholds[row.model])}
|
||||
stroke={COLORS[row.model]} strokeDasharray={row.model === "maai" ? "6 5" : "2 4"} opacity=".55" />
|
||||
<path d={row.path} fill="none" stroke={COLORS[row.model]} strokeWidth={1.6} strokeLinejoin="round" pointerEvents="none" />
|
||||
{row.checkpoints.map(([ms, score]) => <circle key={ms} cx={x(ms)} cy={y(score)} r={3.8}
|
||||
fill="#fffefb" stroke={COLORS[row.model]} strokeWidth={1.8} tabIndex={0} role="button"
|
||||
aria-label={`${row.name} 原标注点评分 ${time(ms)}，${score.toFixed(3)}，点击定位`}
|
||||
onPointerDown={event => event.stopPropagation()}
|
||||
onClick={event => { event.stopPropagation(); selectPoint(ms); }}
|
||||
onKeyDown={event => {
|
||||
if (event.key === "Enter" || event.key === " ") { event.preventDefault(); event.stopPropagation(); selectPoint(ms); }
|
||||
}}><title>{row.name} · 原标注点评分 {score.toFixed(3)} · {time(ms)}</title></circle>)}
|
||||
</g>)}
|
||||
{hoverMs !== null && <line x1={x(hoverMs)} x2={x(hoverMs)} y1={0} y2={SCORE_HEIGHT} stroke="#77816c" opacity=".5" pointerEvents="none" />}
|
||||
</svg>
|
||||
<div className={styles.scoreInspection} aria-hidden="true">
|
||||
<span>{time(inspect)}</span>
|
||||
{series.map(row => {
|
||||
const point = scoreAt(row.points, inspect, row.sample_interval_ms * 2);
|
||||
return <span key={row.model} style={{ color: COLORS[row.model] }}>{row.name} <b>{point ? point[1].toFixed(3) : "—"}</b></span>;
|
||||
})}
|
||||
</div>
|
||||
</div>;
|
||||
}
|
||||
@@ -117,3 +117,37 @@
|
||||
.leftDot, .rightDot { flex-shrink: 0; width: 6px; height: 6px; border-radius: 50%; }
|
||||
.leftDot { background: #6c917c; }
|
||||
.rightDot { background: #658cab; }
|
||||
|
||||
.scorePanelHeader { padding: 0 20px 12px; display: flex; align-items: center; justify-content: space-between; gap: 12px; flex-wrap: wrap; }
|
||||
.scoreToggle { display: inline-flex; align-items: center; gap: 8px; padding: 6px 0; background: none; border: 0; color: #67775b; cursor: pointer; font: inherit; font-size: 11px; }
|
||||
.scoreToggle strong { font-size: 12px; font-weight: 600; color: #45553e; }
|
||||
.scoreToggle > span:last-child { color: #929b89; font-size: 10px; }
|
||||
.chevron, .chevronOpen { display: inline-block; width: 12px; font-size: 23px; line-height: 16px; text-align: center; }
|
||||
.chevronOpen { transform: rotate(90deg); }
|
||||
.scoreControls { display: flex; flex-wrap: wrap; align-items: center; gap: 9px; }
|
||||
.modelControl { display: flex; align-items: center; gap: 6px; padding: 5px 8px; border: 1px solid #e9e9e2; border-radius: 7px; background: #fafbf7; font-size: 10px; }
|
||||
.modelControl strong { font-weight: 600; }
|
||||
.modelControl output { width: 37px; font-family: monospace; font-variant-numeric: tabular-nums; text-align: center; }
|
||||
.modelControl label { display: flex; align-items: center; gap: 4px; color: #88927e; padding-left: 4px; border-left: 1px solid #e2e5db; }
|
||||
.modelControl select { padding: 1px 0; border: 0; background: transparent; font-size: 10px; color: #637258; cursor: pointer; }
|
||||
.modelSwatch { width: 14px; height: 2px; background: currentColor; }
|
||||
.scoreDescription { padding: 0 22px 10px; font-size: 10px; line-height: 1.7; color: #88927f; }
|
||||
.scoreDescription button { border: 0; background: none; color: #536f40; cursor: pointer; text-decoration: underline; font-size: inherit; }
|
||||
.trackNames > .scoreAxis { padding: 12px 8px; flex-direction: column; justify-content: space-between; font-size: 10px; color: #929a87; border-top: 1px solid #e7e9e0; }
|
||||
.scoreAxis span:first-child, .scoreAxis span:last-child { font-family: monospace; align-self: flex-end; }
|
||||
.scoreAxis span:nth-child(2) { color: #647759; }
|
||||
.scoreTrack { position: relative; height: 172px; background: #fdfefb; border-top: 1px solid #e7e9e0; }
|
||||
.scoreTrack > svg { display: block; overflow: hidden; }
|
||||
.scoreTrack circle { cursor: pointer; }
|
||||
.scoreTrack circle:hover, .scoreTrack circle:focus-visible { stroke-width: 3; outline: none; r: 5; }
|
||||
.scoreInspection { position: sticky; left: 10px; margin-top: -26px; width: max-content; max-width: 100%; display: flex; gap: 12px; padding: 4px 7px; border-radius: 4px; color: #8c9682; background: #fffefbe8; font: 10px monospace; pointer-events: none; }
|
||||
.scoreInspection b { font-weight: 500; }
|
||||
.scoreEmpty { position: sticky; left: 0; display: grid; place-items: center; color: #929b88; font-size: 11px; background: #fdfefb; border-top: 1px solid #e7e9e0; }
|
||||
.scoreFootnote { margin: 0; padding: 9px 22px 0; color: #8c9583; font-size: 10px; line-height: 1.7; }
|
||||
.scoreToggle:focus-visible, .modelControl select:focus-visible { outline: 2px solid #758962; outline-offset: 3px; border-radius: 3px; }
|
||||
@media (max-width: 640px) {
|
||||
.scorePanelHeader { padding-inline: 12px; gap: 5px; }
|
||||
.scoreDescription, .scoreFootnote { padding-inline: 12px; }
|
||||
.scoreControls { gap: 5px; }
|
||||
.modelControl { padding: 4px 5px; }
|
||||
}
|
||||
|
||||
@@ -12,6 +12,7 @@ import {
|
||||
} from "react";
|
||||
import styles from "./timeline.module.css";
|
||||
import { resizeTranscriptRange, snapTimelineTime, type SnapPoint } from "@/lib/timeline-edit";
|
||||
import ModelScoreTrack, { ScoreControls, SCORE_HEIGHT, useModelScores, type ScoreThresholds } from "./model-score-track";
|
||||
|
||||
export interface TimelineTranscript {
|
||||
index: number;
|
||||
@@ -29,6 +30,8 @@ export interface TimelineAnnotation {
|
||||
}
|
||||
|
||||
export interface TimelineProps {
|
||||
callId: string;
|
||||
recordingPath: string | null;
|
||||
durationMs: number;
|
||||
currentMs: number;
|
||||
decisionMs: number | null;
|
||||
@@ -66,6 +69,8 @@ function annotationLabel(value: TimelineAnnotation["can_reply_now"], reason?: st
|
||||
}
|
||||
|
||||
export default function Timeline({
|
||||
callId,
|
||||
recordingPath,
|
||||
durationMs,
|
||||
currentMs,
|
||||
decisionMs,
|
||||
@@ -83,6 +88,10 @@ export default function Timeline({
|
||||
onAddAnnotation,
|
||||
}: TimelineProps) {
|
||||
const helpId = useId();
|
||||
const scoreRegionId = useId();
|
||||
const [scoresOpen, setScoresOpen] = useState(false);
|
||||
const [thresholds, setThresholds] = useState<ScoreThresholds>({ maai: .5, smart_turn: .5 });
|
||||
const scores = useModelScores(callId, recordingPath, scoresOpen);
|
||||
const viewportRef = useRef<HTMLDivElement>(null);
|
||||
const canvasRef = useRef<HTMLDivElement>(null);
|
||||
const draggingRef = useRef<
|
||||
@@ -300,7 +309,7 @@ export default function Timeline({
|
||||
|
||||
const channelCount = waveform?.length || 1;
|
||||
const channelLabel = (index: number) => channelCount === 1 ? "单声道" : channelCount === 2 ? index === 0 ? "左声道 L" : "右声道 R" : `声道 ${index + 1}`;
|
||||
const trackStyle = { "--timeline-width": `${canvasWidth}px`, height: 204 + 76 * channelCount } as CSSProperties;
|
||||
const trackStyle = { "--timeline-width": `${canvasWidth}px`, height: 204 + 76 * channelCount + (scoresOpen ? SCORE_HEIGHT : 0) } as CSSProperties;
|
||||
|
||||
return (
|
||||
<section className={styles.timeline} aria-label="对话时间轴">
|
||||
@@ -321,6 +330,20 @@ export default function Timeline({
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className={styles.scorePanelHeader}>
|
||||
<button type="button" className={styles.scoreToggle} aria-expanded={scoresOpen} aria-controls={scoreRegionId}
|
||||
onClick={() => setScoresOpen(value => !value)}>
|
||||
<span aria-hidden="true" className={scoresOpen ? styles.chevronOpen : styles.chevron}>›</span>
|
||||
<strong>模型 EOT 分数</strong><span>{scoresOpen ? "收起" : "展开 MaAI / Smart Turn 对比"}</span>
|
||||
</button>
|
||||
{scoresOpen && <ScoreControls series={scores.series} currentMs={current}
|
||||
thresholds={thresholds} onThreshold={(model, value) => setThresholds(previous => ({ ...previous, [model]: value }))} />}
|
||||
</div>
|
||||
{scoresOpen && <div className={styles.scoreDescription} role="status">
|
||||
{scores.status === "loading" ? "正在读取模型分数…" : scores.status === "error" ? <>{scores.messages.join(" ")} <button type="button" onClick={scores.retry}>重新读取</button></>
|
||||
: <>实线：连续回放评分 · 空心点：原标注点评分 · 虚线：参考阈值。分数越高越倾向接话，MaAI 使用 1 − 用户近期发声分数。{scores.messages.join(" ")}</>}
|
||||
</div>}
|
||||
|
||||
<div className={styles.trackLayout}>
|
||||
<div className={styles.trackNames} aria-hidden="true">
|
||||
<div className={styles.rulerName}>时间</div>
|
||||
@@ -328,6 +351,7 @@ export default function Timeline({
|
||||
<div className={styles.speechName}><span className={styles.agentDot} />机器人</div>
|
||||
<div className={styles.speechName}><span className={styles.userDot} />用户</div>
|
||||
<div className={styles.markerName}>判断点</div>
|
||||
{scoresOpen && <div className={styles.scoreAxis} style={{ height: SCORE_HEIGHT }}><span>1.0</span><span>接话分数</span><span>0.0</span></div>}
|
||||
</div>
|
||||
<div className={styles.viewport} ref={viewportRef}>
|
||||
<div
|
||||
@@ -424,6 +448,12 @@ export default function Timeline({
|
||||
})}
|
||||
</div>
|
||||
|
||||
{scoresOpen && <div id={scoreRegionId}>
|
||||
{scores.series.length ? <ModelScoreTrack series={scores.series} width={canvasWidth} durationMs={duration}
|
||||
currentMs={current} thresholds={thresholds} annotations={annotations} onSeek={onSeek} onSelectAnnotation={onSelectAnnotation} />
|
||||
: <div className={styles.scoreEmpty} style={{ height: SCORE_HEIGHT, width: viewportWidth }}>{scores.status === "loading" ? "正在载入评分曲线…" : "当前录音暂无可显示的模型分数"}</div>}
|
||||
</div>}
|
||||
|
||||
{activeSnap && <div className={styles.snapGuide} style={{ left: position(activeSnap.ms) }} aria-hidden="true" />}
|
||||
<div className={styles.playhead} style={{ left: position(current) }}>
|
||||
<button type="button" className={styles.playheadHandle} aria-label={`拖动播放头 ${timeLabel(current, true)}`}
|
||||
@@ -440,6 +470,8 @@ export default function Timeline({
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{scoresOpen && scores.series.length > 0 && <p className={styles.scoreFootnote}>点击曲线定位录音；悬停查看分数。MaAI 约每 100 ms、Smart Turn 每 200 ms 更新。原标注点评分使用独立历史窗口，与连续曲线可能不同；编辑标注或转写不会重算已有分数。</p>}
|
||||
|
||||
<div className={styles.bottomBar}>
|
||||
<div className={styles.timeReadout}><span className={styles.playheadDot} /> 播放 <strong>{timeLabel(current, true)}</strong></div>
|
||||
<span id={helpId} className={styles.help}>{resizePreview ? "拖动边界调整片段 · 松开保存 · Alt 暂停吸附" : activeSnap ? `已吸附：${activeSnap.label} ${timeLabel(activeSnap.ms, true)}` : "拖动判断点或播放头 · 自动吸附 · Alt 暂停吸附"}</span>
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
export type ModelId = "maai" | "smart_turn";
|
||||
export type ScorePoint = [timeMs: number, score: number];
|
||||
|
||||
export interface ScoreCall {
|
||||
call_id: string;
|
||||
audio_path: string;
|
||||
audio_sha256: string;
|
||||
duration_ms: number;
|
||||
points: ScorePoint[];
|
||||
checkpoints: ScorePoint[];
|
||||
}
|
||||
|
||||
export interface ScoreExport {
|
||||
schema_version: 1;
|
||||
model: ModelId;
|
||||
name: string;
|
||||
score: string;
|
||||
sample_interval_ms: number;
|
||||
calls: ScoreCall[];
|
||||
}
|
||||
|
||||
export interface ScoreSeries extends ScoreCall {
|
||||
model: ModelId;
|
||||
name: string;
|
||||
score: string;
|
||||
sample_interval_ms: number;
|
||||
}
|
||||
|
||||
export interface ScoreResponse {
|
||||
callId: string;
|
||||
recordingPath: string;
|
||||
series: ScoreSeries[];
|
||||
messages: string[];
|
||||
}
|
||||
|
||||
function validPoints(value: unknown, duration: number): value is ScorePoint[] {
|
||||
if (!Array.isArray(value)) return false;
|
||||
let previous = -1;
|
||||
return value.every(point => {
|
||||
if (!Array.isArray(point) || point.length !== 2) return false;
|
||||
const [time, score] = point;
|
||||
if (!Number.isFinite(time) || time < 0 || time > duration || time <= previous
|
||||
|| !Number.isFinite(score) || score < 0 || score > 1) return false;
|
||||
previous = time;
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
/** Validate inference exports without accepting null as a zero score. */
|
||||
export function parseScoreExport(value: unknown, model: ModelId): ScoreExport {
|
||||
const data = value as ScoreExport;
|
||||
if (!data || data.schema_version !== 1 || data.model !== model
|
||||
|| typeof data.name !== "string" || typeof data.score !== "string"
|
||||
|| !Number.isFinite(data.sample_interval_ms) || data.sample_interval_ms <= 0
|
||||
|| !Array.isArray(data.calls)) throw new Error("模型评分文件格式不正确。");
|
||||
const calls = new Set<string>();
|
||||
for (const call of data.calls) {
|
||||
if (!call || typeof call.call_id !== "string" || !call.call_id || calls.has(call.call_id)
|
||||
|| typeof call.audio_path !== "string" || !/^[a-f0-9]{64}$/.test(call.audio_sha256)
|
||||
|| !Number.isFinite(call.duration_ms) || call.duration_ms <= 0
|
||||
|| !validPoints(call.points, call.duration_ms) || !validPoints(call.checkpoints, call.duration_ms)) {
|
||||
throw new Error("模型评分包含重复时间、越界分数或缺失的录音信息。");
|
||||
}
|
||||
calls.add(call.call_id);
|
||||
}
|
||||
return data;
|
||||
}
|
||||
|
||||
/** Only an already available sample may describe a playhead/hover position. */
|
||||
export function scoreAt(points: readonly ScorePoint[], ms: number, maxAgeMs: number): ScorePoint | null {
|
||||
let low = 0, high = points.length - 1, found = -1;
|
||||
while (low <= high) {
|
||||
const mid = (low + high) >>> 1;
|
||||
if (points[mid][0] <= ms) { found = mid; low = mid + 1; } else high = mid - 1;
|
||||
}
|
||||
return found >= 0 && ms - points[found][0] <= maxAgeMs ? points[found] : null;
|
||||
}
|
||||
|
||||
/** Step curves show measured updates; a missing interval starts a new segment. */
|
||||
export function scorePath(points: readonly ScorePoint[], duration: number, width: number,
|
||||
height: number, interval: number): string {
|
||||
const x = (t: number) => (t / duration * width).toFixed(2);
|
||||
const y = (p: number) => (12 + (1 - p) * (height - 24)).toFixed(2);
|
||||
let previous: ScorePoint | null = null;
|
||||
return points.map(point => {
|
||||
const step = previous && point[0] - previous[0] <= interval * 2
|
||||
? `H${x(point[0])}V${y(point[1])}` : `M${x(point[0])},${y(point[1])}`;
|
||||
previous = point;
|
||||
return step;
|
||||
}).join("");
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
import assert from "node:assert/strict";
|
||||
import test from "node:test";
|
||||
import { readFile } from "node:fs/promises";
|
||||
// @ts-expect-error Node's native TypeScript runner resolves explicit .ts imports.
|
||||
import { parseScoreExport, scoreAt, scorePath } from "../lib/model-scores.ts";
|
||||
|
||||
const fixture = () => ({ schema_version: 1, model: "maai", name: "MaAI", score: "1-p_now[user]", sample_interval_ms: 100,
|
||||
calls: [{ call_id: "demo", audio_path: "demo.mp3", audio_sha256: "a".repeat(64), duration_ms: 1000,
|
||||
points: [[100, .2], [200, .9], [900, .3]], checkpoints: [[150, .8]] }] });
|
||||
|
||||
test("model scores accept real ordered samples and independent checkpoint times", () => {
|
||||
assert.equal(parseScoreExport(fixture(), "maai").calls[0].checkpoints[0][0], 150);
|
||||
});
|
||||
|
||||
test("invalid exports cannot masquerade as zero scores or another model", () => {
|
||||
for (const points of [[[100, null]], [[100, 1.1]], [[100, -1]], [[100, .5], [100, .6]], [[200, .5], [100, .6]], [[1001, .5]]]) {
|
||||
const data = fixture(); data.calls[0].points = points as number[][];
|
||||
assert.throws(() => parseScoreExport(data, "maai"));
|
||||
}
|
||||
assert.throws(() => parseScoreExport(fixture(), "smart_turn"));
|
||||
const duplicate = fixture(); duplicate.calls.push(duplicate.calls[0]);
|
||||
assert.throws(() => parseScoreExport(duplicate, "maai"));
|
||||
});
|
||||
|
||||
test("playhead values never use a future sample or carry through missing data", () => {
|
||||
const points: [number, number][] = [[100, .2], [200, .9], [900, .3]];
|
||||
assert.equal(scoreAt(points, 99, 200), null);
|
||||
assert.deepEqual(scoreAt(points, 150, 200), [100, .2]);
|
||||
assert.deepEqual(scoreAt(points, 200, 200), [200, .9]);
|
||||
assert.equal(scoreAt(points, 600, 200), null);
|
||||
assert.equal(scoreAt([], 100, 200), null);
|
||||
});
|
||||
|
||||
test("curve gaps are not connected and steps remain on the audio clock", () => {
|
||||
const path = scorePath([[100, .2], [200, .9], [900, .3]], 1000, 500, 100, 100);
|
||||
assert.equal(path, "M50.00,72.80H100.00V19.60M450.00,65.20");
|
||||
});
|
||||
|
||||
test("recorded inference exports agree on audio identity and retain all 44 checkpoints", async () => {
|
||||
const files = [
|
||||
["maai", "maai-ch-kyoto"], ["smart_turn", "smart-turn-v3.2"],
|
||||
] as const;
|
||||
const exports = await Promise.all(files.map(async ([model, directory]) => parseScoreExport(
|
||||
JSON.parse(await readFile(new URL(`../../../experiments/${directory}/results/score-timeline.json`, import.meta.url), "utf8")), model)));
|
||||
const identity = (data: typeof exports[number]) => data.calls.map(call => [call.call_id, call.audio_path, call.audio_sha256, call.duration_ms]);
|
||||
assert.deepEqual(identity(exports[0]), identity(exports[1]));
|
||||
for (const data of exports) {
|
||||
assert.equal(data.calls.reduce((n, call) => n + call.checkpoints.length, 0), 44);
|
||||
for (const call of data.calls) {
|
||||
assert.ok(call.points.length > 100);
|
||||
assert.ok(call.points.at(-1)![0] <= call.duration_ms);
|
||||
}
|
||||
}
|
||||
});
|
||||
Reference in New Issue
Block a user