wx44wx/xiaobailing-turn-detection
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

新增 Silero VAD 时间轴评分对比与可复现推理

Browse Source
This commit is contained in:
Xin Wang
2026-10-22 21:25:47 +08:00
parent 43724c13e4
commit c9b618e634
14 changed files with 379 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
labels/eot-studio/README.md
+5 -4
View File
@@ -24,15 +24,16 @@ npm run dev
浏览器草稿自动保留，刷新后可恢复。网页保存不覆盖原 XLSX、初稿和录音。初稿转换方法、标签映射与限制见 [转换说明](../EOT标注/韩泰初稿/转换说明.md)，原行号及原结论保留在初稿目录的 `source_rows.json` 中。原表没有时间信息，录音中未收录的开场与其他发言需人工补充。
## 模型 EOT 分数
## 模型分数对比
点击时间轴上方的“模型 EOT 分数”展开对比，默认收起。MaAI（紫色）和 Smart Turn（青色）共用录音的时间刻度、缩放、播放头及判断点参考线。点击或拖动曲线区域定位播放，悬停查看当时已产生的分数；点击空心评分点定位到原标注时刻。顶部显示播放位置最近的有效评分，暂无评分时显示“—”，不将缺失数据当作 0。
点击时间轴上方的“模型分数对比”展开对比，默认收起。VAP / MaAI（紫色）、Smart Turn（青色）和 Silero VAD（橙色）共用录音的时间刻度、缩放、播放头及判断点参考线。点击或拖动曲线区域定位播放，悬停查看当时已产生的分数；点击空心评分点定位到原标注时刻。顶部显示播放位置最近的有效评分，暂无评分时显示“—”，不将缺失数据当作 0。
- 实线阶梯曲线来自对六通完整录音的真实推理，不是用44个点插值。MaAI每100ms输入一块音频，因流式重采样缓冲，部分评分间隔为200ms；Smart Turn每200ms计算一次，并包含原标注的精确时刻。
- MaAI使用双声道连续回放、20秒KV缓存，分数为 `1 - p_now[user]`，是接话候选分数，不是校准的EOT概率。Smart Turn沿用原回测的用户声道、过去ASR定位、500ms前置音频、3秒内续说及最多8秒窗口；没有额外的结束信号、VAD停止门槛或超时放行。
- 空心点保留原44点的独立窗口回测分数。尤其MaAI的连续状态与独立窗口不同，空心点可能不落在连续曲线上。原始44点指标不会被这次时间序列覆盖。
- 参考阈值只改变虚线位置，不修改人工标签。MaAI可选0.11、0.5、0.8，Smart Turn可选0.5、0.8。保存的分数不会随标注或转写编辑而重新计算。
- 两个模型名称旁各有独立显示开关，默认都打开。关闭后隐藏该模型的曲线、原标注点评分、参考线及分数读数；重新打开保留原阈值。两个都关闭时显示提示，切换通话或收起再展开会保留当前显示选择。
- Silero VAD 为用户声道逐个完整 32 ms 帧的真实 ONNX 推理，使用 `1 - speech_probability`，仅为非发声基线，不是语义 EOT 概率。时间戳为帧末；空心点取原判断时刻最后可用帧，不读取未来音频。独立环境和复现方法见 [Silero 实验说明](../../experiments/silero-vad/README.md)。
- 参考阈值只改变虚线位置，不修改人工标签。MaAI可选0.11、0.5、0.8，Smart Turn和Silero可选0.5、0.8。保存的分数不会随标注或转写编辑而重新计算。
- 三个模型名称旁各有独立显示开关，默认都打开。关闭后隐藏该模型的曲线、原标注点评分、参考线及分数读数；重新打开保留原阈值。全部关闭时显示提示，切换通话或收起再展开会保留当前显示选择。
- 仅在展开时读取当前项目录音的评分，并核对音频SHA256。更换为本地新录音、音频变化或没有对应推理数据时，不显示旧曲线。切换通话会取消旧请求，避免前一段数据覆盖当前段。
评分数据位于各模型实验目录的 `results/score-timeline.json`。更新数据时，在两个各自的本机uv环境中运行：
labels/eot-studio/app/api/model-scores/route.ts
+1
View File
@@ -9,6 +9,7 @@ const root = path.resolve(process.cwd(), "..", "..");
const sources: { model: ModelId; directory: string; name: string }[] = [
{ model: "maai", directory: "maai-ch-kyoto", name: "MaAI" },
{ model: "smart_turn", directory: "smart-turn-v3.2", name: "Smart Turn" },
{ model: "silero_vad", directory: "silero-vad", name: "Silero VAD" },
];
const json = (data: unknown, status = 200) => Response.json(data, { status, headers: { "Cache-Control": "no-store" } });
labels/eot-studio/components/model-score-track.tsx
+2 -2
View File
@@ -6,7 +6,7 @@ import type { TimelineAnnotation } from "./timeline";
import styles from "./timeline.module.css";
const EMPTY: ScoreSeries[] = [];
const COLORS: Record<ModelId, string> = { maai: "#8866b2", smart_turn: "#278d95" };
const COLORS: Record<ModelId, string> = { maai: "#8866b2", smart_turn: "#278d95", silero_vad: "#c47b32" };
export const SCORE_HEIGHT = 172;
export type ScoreThresholds = Record<ModelId, number>;
export type ScoreVisibility = Record<ModelId, boolean>;
@@ -86,7 +86,7 @@ export default function ModelScoreTrack({ series, width, durationMs, currentMs,
};
return <div className={styles.scoreTrack} onPointerLeave={() => setHoverMs(null)}>
<svg width={width} height={SCORE_HEIGHT} viewBox={`0 0 ${width} ${SCORE_HEIGHT}`}
role="group" aria-label="MaAI 与 Smart Turn 的 EOT 分数，纵轴 0 到 1，横轴与录音时间一致"
role="group" aria-label="VAP、Smart Turn 与 Silero VAD 分数对比，纵轴 0 到 1，横轴与录音时间一致"
onPointerMove={event => {
const bounds = event.currentTarget.getBoundingClientRect();
setHoverMs(Math.max(0, Math.min(durationMs, (event.clientX - bounds.left) / bounds.width * durationMs)));
labels/eot-studio/components/timeline.tsx
+6 -6
View File
@@ -90,8 +90,8 @@ export default function Timeline({
const helpId = useId();
const scoreRegionId = useId();
const [scoresOpen, setScoresOpen] = useState(false);
const [thresholds, setThresholds] = useState<ScoreThresholds>({ maai: .5, smart_turn: .5 });
const [scoreVisibility, setScoreVisibility] = useState<ScoreVisibility>({ maai: true, smart_turn: true });
const [thresholds, setThresholds] = useState<ScoreThresholds>({ maai: .5, smart_turn: .5, silero_vad: .5 });
const [scoreVisibility, setScoreVisibility] = useState<ScoreVisibility>({ maai: true, smart_turn: true, silero_vad: true });
const scores = useModelScores(callId, recordingPath, scoresOpen);
const visibleScores = useMemo(() => scores.series.filter(row => scoreVisibility[row.model]), [scores.series, scoreVisibility]);
const viewportRef = useRef<HTMLDivElement>(null);
@@ -336,7 +336,7 @@ export default function Timeline({
<button type="button" className={styles.scoreToggle} aria-expanded={scoresOpen} aria-controls={scoreRegionId}
onClick={() => setScoresOpen(value => !value)}>
<span aria-hidden="true" className={scoresOpen ? styles.chevronOpen : styles.chevron}>›</span>
<strong>模型 EOT 分数</strong><span>{scoresOpen ? "收起" : "展开 MaAI / Smart Turn 对比"}</span>
<strong>模型分数对比</strong><span>{scoresOpen ? "收起" : "展开 VAP / Smart Turn / Silero VAD 对比"}</span>
</button>
{scoresOpen && <ScoreControls series={scores.series} currentMs={current}
visibility={scoreVisibility} onVisibility={(model, visible) => setScoreVisibility(previous => ({ ...previous, [model]: visible }))}
@@ -344,7 +344,7 @@ export default function Timeline({
</div>
{scoresOpen && <div className={styles.scoreDescription} role="status">
{scores.status === "loading" ? "正在读取模型分数…" : scores.status === "error" ? <>{scores.messages.join(" ")} <button type="button" onClick={scores.retry}>重新读取</button></>
: <>实线：连续回放评分 · 空心点：原标注点评分 · 虚线：参考阈值。分数越高越倾向接话，MaAI 使用 1 − 用户近期发声分数。{scores.messages.join(" ")}</>}
: <>实线：连续回放评分 · 空心点：原标注点评分 · 虚线：参考阈值。Smart Turn 表示完成倾向；VAP 使用 1 − 用户近期发声分数，Silero 使用 1 − 当前发声分数，仅为非发声基线，不是语义 EOT 概率。{scores.messages.join(" ")}</>}
</div>}
<div className={styles.trackLayout}>
@@ -354,7 +354,7 @@ export default function Timeline({
<div className={styles.speechName}><span className={styles.agentDot} />机器人</div>
<div className={styles.speechName}><span className={styles.userDot} />用户</div>
<div className={styles.markerName}>判断点</div>
{scoresOpen && <div className={styles.scoreAxis} style={{ height: SCORE_HEIGHT }}><span>1.0</span><span>接话分数</span><span>0.0</span></div>}
{scoresOpen && <div className={styles.scoreAxis} style={{ height: SCORE_HEIGHT }}><span>1.0</span><span>对比分数</span><span>0.0</span></div>}
</div>
<div className={styles.viewport} ref={viewportRef}>
<div
@@ -473,7 +473,7 @@ export default function Timeline({
</div>
</div>
{scoresOpen && scores.series.length > 0 && <p className={styles.scoreFootnote}>点击曲线定位录音；悬停查看分数。MaAI 约每 100 ms、Smart Turn 每 200 ms 更新。原标注点评分使用独立历史窗口，与连续曲线可能不同；编辑标注或转写不会重算已有分数。</p>}
{scoresOpen && scores.series.length > 0 && <p className={styles.scoreFootnote}>点击曲线定位录音；悬停查看分数。VAP 约每 100 ms、Smart Turn 每 200 ms、Silero 每 32 ms 更新。Silero 空心点取判断前最后完整帧；其他模型原标注点评分使用独立历史窗口，与连续曲线可能不同；编辑标注或转写不会重算已有分数。</p>}
<div className={styles.bottomBar}>
<div className={styles.timeReadout}><span className={styles.playheadDot} /> 播放 <strong>{timeLabel(current, true)}</strong></div>
labels/eot-studio/lib/model-scores.ts
+1 -1
View File
@@ -1,4 +1,4 @@
export type ModelId = "maai" | "smart_turn";
export type ModelId = "maai" | "smart_turn" | "silero_vad";
export type ScorePoint = [timeMs: number, score: number];
export interface ScoreCall {
labels/eot-studio/tests/model-scores.test.ts
+18 -1
View File
@@ -38,12 +38,13 @@ test("curve gaps are not connected and steps remain on the audio clock", () => {
test("recorded inference exports agree on audio identity and retain all 44 checkpoints", async () => {
const files = [
["maai", "maai-ch-kyoto"], ["smart_turn", "smart-turn-v3.2"],
["maai", "maai-ch-kyoto"], ["smart_turn", "smart-turn-v3.2"], ["silero_vad", "silero-vad"],
] as const;
const exports = await Promise.all(files.map(async ([model, directory]) => parseScoreExport(
JSON.parse(await readFile(new URL(`../../../experiments/${directory}/results/score-timeline.json`, import.meta.url), "utf8")), model)));
const identity = (data: typeof exports[number]) => data.calls.map(call => [call.call_id, call.audio_path, call.audio_sha256, call.duration_ms]);
assert.deepEqual(identity(exports[0]), identity(exports[1]));
assert.deepEqual(identity(exports[0]), identity(exports[2]));
for (const data of exports) {
assert.equal(data.calls.reduce((n, call) => n + call.checkpoints.length, 0), 44);
for (const call of data.calls) {
@@ -52,3 +53,19 @@ test("recorded inference exports agree on audio identity and retain all 44 check
}
}
});
test("Silero checkpoints use only the last available frame, and speech is inverted once", async () => {
const data = parseScoreExport(JSON.parse(await readFile(new URL(
"../../../experiments/silero-vad/results/score-timeline.json", import.meta.url), "utf8")), "silero_vad");
assert.equal(data.sample_interval_ms, 32);
for (const call of data.calls) {
assert.equal(call.points[0][0], 32);
for (const [ms] of call.points) assert.equal(ms % 32, 0);
for (const [ms, score] of call.checkpoints) {
const frame = scoreAt(call.points, ms, 32);
assert.ok(frame);
assert.equal(frame[1], score);
assert.ok(frame[0] <= ms && ms - frame[0] < 32);
}
}
});