接入 LiveKit 与 TurnSense 评分并改进模型曲线选择

2026-10-22 23:52:16 +08:00
parent c9b618e634
commit 19d4e3c767
38 changed files with 2652 additions and 58 deletions
@@ -0,0 +1,45 @@
+call_id,decision_ms,can_reply_now,probability,prediction,threshold,input_samples,available_end_ms,preparation_ms,inference_ms
+2c83253a-7977-4e3c-a387-0aa1ea06fbe1,21780,1,0.4333016276359558,1,0.355,19200,21780.0,2.7287499979138374,5.575624294579029
+2c83253a-7977-4e3c-a387-0aa1ea06fbe1,29860,1,0.6237799525260925,1,0.355,19200,29860.0,3.7509994581341743,6.744083017110825
+2c83253a-7977-4e3c-a387-0aa1ea06fbe1,48125,0,0.45254063606262207,1,0.355,19200,48125.0,5.758457817137241,7.209542207419872
+2c83253a-7977-4e3c-a387-0aa1ea06fbe1,64000,1,0.2848171889781952,0,0.355,19200,64000.0,7.960332557559013,6.5420838072896
+2c83253a-7977-4e3c-a387-0aa1ea06fbe1,76672,0,0.45449426770210266,1,0.355,19200,76672.0,9.708958677947521,7.211333140730858
+2c83253a-7977-4e3c-a387-0aa1ea06fbe1,88482,1,0.45394039154052734,1,0.355,19200,88482.0,11.292457580566406,5.600791424512863
+40ae686c-8196-4d7b-b7ad-3e12a427ebb6,22007,1,0.5854928493499756,1,0.355,19200,22007.0,2.911125309765339,8.196083828806877
+40ae686c-8196-4d7b-b7ad-3e12a427ebb6,31842,1,0.5878404378890991,1,0.355,19200,31842.0,4.0603335946798325,5.483958870172501
+40ae686c-8196-4d7b-b7ad-3e12a427ebb6,48536,0,0.2537097632884979,0,0.355,19200,48536.0,5.920499563217163,5.44358417391777
+40ae686c-8196-4d7b-b7ad-3e12a427ebb6,52348,1,0.557003378868103,1,0.355,19200,52348.0,6.229749880731106,7.242332212626934
+40ae686c-8196-4d7b-b7ad-3e12a427ebb6,65298,0,0.6249343752861023,1,0.355,19200,65298.0,7.885375060141087,6.663625128567219
+40ae686c-8196-4d7b-b7ad-3e12a427ebb6,67943,0,0.6181272268295288,1,0.355,19200,67943.0,8.195959031581879,7.16041773557663
+40ae686c-8196-4d7b-b7ad-3e12a427ebb6,70430,0,0.2462766468524933,0,0.355,19200,70430.0,8.394083939492702,6.635541096329689
+40ae686c-8196-4d7b-b7ad-3e12a427ebb6,82941,1,0.5786489844322205,1,0.355,19200,82941.0,10.166166350245476,6.077749654650688
+872a1e67-873a-4c49-a598-b38a670afcad,17153,1,0.5009272694587708,1,0.355,19200,17153.0,2.16870941221714,6.159832701086998
+872a1e67-873a-4c49-a598-b38a670afcad,24846,1,0.668940544128418,1,0.355,19200,24846.0,3.041500225663185,5.271041765809059
+872a1e67-873a-4c49-a598-b38a670afcad,33471,1,0.6602025628089905,1,0.355,19200,33471.0,4.0121255442500114,6.265167146921158
+872a1e67-873a-4c49-a598-b38a670afcad,44810,0,0.16742786765098572,0,0.355,19200,44810.0,5.381416529417038,7.350374944508076
+872a1e67-873a-4c49-a598-b38a670afcad,45581,0,0.3364622890949249,0,0.355,19200,45581.0,5.505583249032497,5.54287526756525
+872a1e67-873a-4c49-a598-b38a670afcad,47187,0,0.3902971148490906,1,0.355,19200,47187.0,5.704415962100029,6.308957934379578
+872a1e67-873a-4c49-a598-b38a670afcad,49500,1,0.6626229882240295,1,0.355,19200,49500.0,5.960375070571899,6.991500034928322
+872a1e67-873a-4c49-a598-b38a670afcad,52326,0,0.2527875006198883,0,0.355,19200,52326.0,6.491790525615215,5.456083454191685
+872a1e67-873a-4c49-a598-b38a670afcad,54318,1,0.14241522550582886,0,0.355,19200,54318.0,6.935875862836838,5.50837442278862
+a266b9ef-5e47-4434-a04c-ce4eacd52c19,18027,1,0.6745697259902954,1,0.355,19200,18027.0,2.313540317118168,7.023083046078682
+a266b9ef-5e47-4434-a04c-ce4eacd52c19,24768,1,0.6706604957580566,1,0.355,19200,24768.0,3.0613746494054794,5.433417856693268
+a266b9ef-5e47-4434-a04c-ce4eacd52c19,41219,0,0.5743900537490845,1,0.355,19200,41219.0,5.006374791264534,6.470665335655212
+a266b9ef-5e47-4434-a04c-ce4eacd52c19,51080,0,0.47520971298217773,1,0.355,19200,51080.0,6.177124567329884,7.069415412843227
+a266b9ef-5e47-4434-a04c-ce4eacd52c19,68313,0,0.6697428226470947,1,0.355,19200,68313.0,8.281541988253593,7.906915619969368
+cb3238dc-3946-42a5-aad8-9b9a54226363,13540,1,0.5981559753417969,1,0.355,19200,13540.0,1.6896668821573257,5.790833383798599
+cb3238dc-3946-42a5-aad8-9b9a54226363,19620,1,0.651459813117981,1,0.355,19200,19620.0,2.6979586109519005,5.748957395553589
+cb3238dc-3946-42a5-aad8-9b9a54226363,27520,1,0.6795960068702698,1,0.355,19200,27520.0,3.496958874166012,5.925584584474564
+cb3238dc-3946-42a5-aad8-9b9a54226363,41207,0,0.6371078491210938,1,0.355,19200,41207.0,5.00095821917057,5.397791974246502
+cb3238dc-3946-42a5-aad8-9b9a54226363,49204,0,0.6755564212799072,1,0.355,19200,49204.0,6.174792535603046,5.501791834831238
+cb3238dc-3946-42a5-aad8-9b9a54226363,58424,0,0.6663632392883301,1,0.355,19200,58424.0,7.433624938130379,5.379541777074337
+cb3238dc-3946-42a5-aad8-9b9a54226363,60790,0,0.25056231021881104,0,0.355,19200,60790.0,7.39095825701952,6.392749957740307
+cb3238dc-3946-42a5-aad8-9b9a54226363,70745,0,0.5162182450294495,1,0.355,19200,70745.0,8.840749971568584,5.87591715157032
+cb3238dc-3946-42a5-aad8-9b9a54226363,73640,0,0.387985497713089,1,0.355,19200,73640.0,8.819833397865295,5.357082933187485
+e304f740-85d4-427e-9fd6-c5474e9cd737,11531,1,0.6718385219573975,1,0.355,19200,11531.0,1.627875491976738,5.708709359169006
+e304f740-85d4-427e-9fd6-c5474e9cd737,18842,1,0.5978389978408813,1,0.355,19200,18842.0,2.3520831018686295,6.592665798962116
+e304f740-85d4-427e-9fd6-c5474e9cd737,29619,1,0.6137039065361023,1,0.355,19200,29619.0,3.6592911928892136,6.604207679629326
+e304f740-85d4-427e-9fd6-c5474e9cd737,40701,0,0.6689900755882263,1,0.355,19200,40701.0,4.9554575234651566,5.339334718883038
+e304f740-85d4-427e-9fd6-c5474e9cd737,54600,1,0.6682158708572388,1,0.355,19200,54600.0,6.610791198909283,6.827791221439838
+e304f740-85d4-427e-9fd6-c5474e9cd737,62828,0,0.7292628288269043,1,0.355,19200,62828.0,7.489999756217003,5.283292382955551
+e304f740-85d4-427e-9fd6-c5474e9cd737,65819,0,0.6575707197189331,1,0.355,19200,65819.0,8.478624746203423,5.405125208199024
@@ -0,0 +1,8 @@
+WITNESS CPU seeded probability=0.580093086, repeat=0.580093086, parallelism=2
+2c83253a: 489 scores, 6 checkpoints; repeat/future-suffix checks passed
+40ae686c: 441 scores, 8 checkpoints; repeat/future-suffix checks passed
+872a1e67: 379 scores, 9 checkpoints; repeat/future-suffix checks passed
+a266b9ef: 421 scores, 5 checkpoints; repeat/future-suffix checks passed
+cb3238dc: 478 scores, 9 checkpoints; repeat/future-suffix checks passed
+e304f740: 365 scores, 7 checkpoints; repeat/future-suffix checks passed
+{"threshold": 0.355, "n": 44, "tn": 6, "fp": 16, "fn": 2, "tp": 20, "accuracy": 0.5909090909090909, "false_reply_rate": 0.7272727272727273, "false_hold_rate": 0.09090909090909091}
@@ -0,0 +1,2 @@
+WITNESS CPU seeded probability=0.580093086, repeat=0.580093086, parallelism=2
+{"load_ms": 123.47654160112143, "provenance": {"machine": "macOS-26.4.1-arm64-arm-64bit", "hardware": "Apple M4 Mac mini CPU, 16 GB", "python": "3.11.15 (main, Mar 25 2026, 02:58:47) [Clang 22.1.1 ]", "dependencies": {"livekit-local-inference": "0.2.7", "livekit-agents": "1.8.8", "numpy": "2.2.6", "scipy": "1.15.3", "soundfile": "0.13.1"}, "native_binary_sha256": "7afa5899367957328c9916153b5d672392e191c0bfadf1f49325af2e4f3055b4", "weight_identity": "Weights embedded in pinned official binary; separate hash not exposed", "wheel_sha256": "fa4cb2d495c80be6121642dba4def3a88b22a68d52c1b0affccd66c563de6622", "native_parallelism": 2, "script_sha256": "e05a0408fcce2a8194966bf594d7324e17cd8025d0b7eb06152e58eb9b46f892", "annotation_sha256": "780e50f976daf039ab976ea155bdb5d1ca5fffc792f9db7adf8063ec3048ff8e"}}
@@ -0,0 +1,60 @@
+{
+  "metrics": {
+    "threshold": 0.355,
+    "n": 44,
+    "tn": 6,
+    "fp": 16,
+    "fn": 2,
+    "tp": 20,
+    "accuracy": 0.5909090909090909,
+    "false_reply_rate": 0.7272727272727273,
+    "false_hold_rate": 0.09090909090909091
+  },
+  "reference_threshold_05": {
+    "threshold": 0.5,
+    "n": 44,
+    "tn": 11,
+    "fp": 11,
+    "fn": 4,
+    "tp": 18,
+    "accuracy": 0.6590909090909091,
+    "false_reply_rate": 0.5,
+    "false_hold_rate": 0.18181818181818182
+  },
+  "model": "LiveKit Turn Detector v1-mini",
+  "official_zh_threshold": 0.355,
+  "load_ms": 119.18720882385969,
+  "method": "Agents 1.8.8 local transport, native v1-mini 0.2.7. User channel 0; received int16 prefix /32768 cropped BEFORE scipy resample_poly to 16kHz; last 19200 samples converted with *32767. Native left-padding below 1.2s. No ASR, VAD gate, silence timeout or fallback scores. EOT_PARALLELISM=1. Prefix resampling differs from official librosa and online streaming resampling; not a full service replay.",
+  "provenance": {
+    "machine": "macOS-26.4.1-arm64-arm-64bit",
+    "hardware": "Apple M4 Mac mini CPU, 16 GB",
+    "python": "3.11.15 (main, Mar 25 2026, 02:58:47) [Clang 22.1.1 ]",
+    "dependencies": {
+      "livekit-local-inference": "0.2.7",
+      "livekit-agents": "1.8.8",
+      "numpy": "2.2.6",
+      "scipy": "1.15.3",
+      "soundfile": "0.13.1"
+    },
+    "native_binary_sha256": "7afa5899367957328c9916153b5d672392e191c0bfadf1f49325af2e4f3055b4",
+    "weight_identity": "Weights embedded in pinned official binary; separate hash not exposed",
+    "wheel_sha256": "fa4cb2d495c80be6121642dba4def3a88b22a68d52c1b0affccd66c563de6622",
+    "native_parallelism": 2,
+    "script_sha256": "e05a0408fcce2a8194966bf594d7324e17cd8025d0b7eb06152e58eb9b46f892",
+    "annotation_sha256": "780e50f976daf039ab976ea155bdb5d1ca5fffc792f9db7adf8063ec3048ff8e"
+  },
+  "preparation_ms": {
+    "p50": 5.839478690177202,
+    "p95": 9.578727371990682
+  },
+  "inference_ms": {
+    "p50": 6.118791177868843,
+    "p95": 7.3341685347259045
+  },
+  "limitations": [
+    "44 selected points: exploratory, not held-out generalization.",
+    "Point false reply rate is not interval interruption rate; user channel may contain robot echo.",
+    "Raw zh threshold without production VAD/endpointing gates.",
+    "Wheel license restricts use to LiveKit Agents; do not use outputs to improve other models."
+  ]
+}
@@ -0,0 +1,16 @@
+# LiveKit v1-mini 韩泰评估结果
+
+本机 M4 CPU；6 通录音、44 个判断点，正负各 22。官方中文阈值来自锁定 SDK 的 LOCAL_LANGUAGES[zh]，未对本批数据搜索最优阈值。
+
+| 阈值 | 正确 / 总数 | 准确率 | 误接话（应等待却接话） | 漏接话（应接话却等待） |
+| --- | --- | --- | --- | --- |
+| 0.355 | 26 / 44 | 59.09% | 16 / 22（72.73%） | 2 / 22（9.09%） |
+| 0.5 | 29 / 44 | 65.91% | 11 / 22（50.00%） | 4 / 22（18.18%） |
+
+模型已加载后，框架本地推理调用（包含异步派发）中位数 6.12 ms、P95 7.33 ms；输入准备/前缀重采样中位数 5.84 ms、P95 9.58 ms。模型构造耗时 119.19 ms，不含 Python/Agents 导入，不是完整进程冷启动时间。原生库报告 parallelism=2，不能称为单线程。耗时不等于端点等待时间。
+
+连续曲线每 200 ms 独立计算最近 1.2 秒窗口，并包含精确判断点，非插值生成。全部 44 点验证了重复结果一致，以及修改未来音频不改变输入。本实验读取当前标注；已有其他模型结果来自之前的标注快照，已发现第二通录音有一个判断点从旧结果的 48348 ms 移至当前 48536 ms，其余 43 点时间和标签一致。因此不能直接拿旧 44 点汇总作严格横向排名；旧曲线空心点仍保留原快照时刻。
+
+中文默认阈值下，16 个应等待点误判，仍不能认为解决了韩泰抢话问题。0.5 是预先选定参考值，不是独立验证的最优阈值。小样本手选点仅为探索性回测，点级误接话率不等于连续对话打断率，用户声道可能有机器人回声。
+
+只评估原始模型分数，不含 LiveKit 会话静音、VAD、超时或故障放行策略。预处理采用判断前裁取再重采样，具体差异见上级 README。模型许可限制见上级 README，结果不用于其他模型训练。