接入 LiveKit 与 TurnSense 评分并改进模型曲线选择
This commit is contained in:
Xin Wang
@@ -0,0 +1,45 @@
|
||||
call_id,decision_ms,can_reply_now,p_complete,p_incomplete,p_invalid,native_class,native_can_reply,prediction_05,input_samples,available_end_ms,preparation_ms,frontend_ms,inference_ms
|
||||
2c83253a-7977-4e3c-a387-0aa1ea06fbe1,21780,1,0.7006610631942749,0.29899370670318604,0.00034517815220169723,complete,1,1,128000,21780.0,0.6559574976563454,4.819874651730061,31.615042127668858
|
||||
2c83253a-7977-4e3c-a387-0aa1ea06fbe1,29860,1,0.6634299755096436,0.3349137306213379,0.0016562796663492918,complete,1,1,128000,29860.0,1.0954178869724274,5.1015829667449,32.756125554442406
|
||||
2c83253a-7977-4e3c-a387-0aa1ea06fbe1,48125,0,0.37549158930778503,0.624477744102478,3.058241418329999e-05,incomplete,0,0,128000,48125.0,1.650916412472725,5.288207903504372,31.92400000989437
|
||||
2c83253a-7977-4e3c-a387-0aa1ea06fbe1,64000,1,0.48844802379608154,0.5109370946884155,0.0006149054388515651,incomplete,0,0,128000,64000.0,2.638166770339012,5.002083256840706,35.11762525886297
|
||||
2c83253a-7977-4e3c-a387-0aa1ea06fbe1,76672,0,0.13564634323120117,0.8621996641159058,0.0021539577282965183,incomplete,0,0,128000,76672.0,3.0605001375079155,5.475125275552273,30.60787543654442
|
||||
2c83253a-7977-4e3c-a387-0aa1ea06fbe1,88482,1,0.942379891872406,0.05571956932544708,0.0019004124915227294,complete,1,1,128000,88482.0,3.272208385169506,4.725124686956406,35.046874545514584
|
||||
40ae686c-8196-4d7b-b7ad-3e12a427ebb6,22007,1,0.3481478691101074,0.651701033115387,0.00015111359243746847,incomplete,0,0,128000,22007.0,0.7836250588297844,4.815500229597092,31.588083133101463
|
||||
40ae686c-8196-4d7b-b7ad-3e12a427ebb6,31842,1,0.4427306354045868,0.5568790435791016,0.000390340865124017,incomplete,0,0,128000,31842.0,0.8989162743091583,4.637166857719421,31.384666450321674
|
||||
40ae686c-8196-4d7b-b7ad-3e12a427ebb6,48536,0,0.7467879056930542,0.2531856596469879,2.6438898203196004e-05,complete,1,1,128000,48536.0,1.652708277106285,4.660000093281269,29.796875081956387
|
||||
40ae686c-8196-4d7b-b7ad-3e12a427ebb6,52348,1,0.9037413597106934,0.09623286128044128,2.5730008928803727e-05,complete,1,1,128000,52348.0,2.6957085356116295,6.087250076234341,32.25929103791714
|
||||
40ae686c-8196-4d7b-b7ad-3e12a427ebb6,65298,0,0.16583986580371857,0.8341189026832581,4.125330087845214e-05,incomplete,0,0,128000,65298.0,2.300082705914974,4.840708337724209,29.82383221387863
|
||||
40ae686c-8196-4d7b-b7ad-3e12a427ebb6,67943,0,0.12022174149751663,0.8796190023422241,0.00015931774396449327,incomplete,0,0,128000,67943.0,2.689083106815815,5.102500319480896,31.059750355780125
|
||||
40ae686c-8196-4d7b-b7ad-3e12a427ebb6,70430,0,0.0029643976595252752,0.9970191717147827,1.6417250662925653e-05,incomplete,0,0,128000,70430.0,2.258916385471821,4.696916788816452,31.458708457648754
|
||||
40ae686c-8196-4d7b-b7ad-3e12a427ebb6,82941,1,0.5291794538497925,0.4702177941799164,0.0006027779309079051,complete,1,1,128000,82941.0,3.3505000174045563,5.25750033557415,31.063792295753956
|
||||
872a1e67-873a-4c49-a598-b38a670afcad,17153,1,0.6752944588661194,0.32434993982315063,0.0003555795119609684,complete,1,1,128000,17153.0,0.5732094869017601,4.804333671927452,31.661707907915115
|
||||
872a1e67-873a-4c49-a598-b38a670afcad,24846,1,0.19226661324501038,0.8076792359352112,5.4253305279416963e-05,incomplete,0,0,128000,24846.0,0.7762927561998367,5.662750452756882,31.10175020992756
|
||||
872a1e67-873a-4c49-a598-b38a670afcad,33471,1,0.670210599899292,0.3295625150203705,0.00022692007769364864,complete,1,1,128000,33471.0,1.130000688135624,4.772042855620384,31.292874366044998
|
||||
872a1e67-873a-4c49-a598-b38a670afcad,44810,0,0.613646924495697,0.3845664858818054,0.0017866046400740743,complete,1,1,128000,44810.0,1.7957082018256187,5.095167085528374,30.916291289031506
|
||||
872a1e67-873a-4c49-a598-b38a670afcad,45581,0,0.5043333172798157,0.4948233664035797,0.0008433183538727462,complete,1,1,128000,45581.0,1.8424158915877342,4.733665846288204,31.039416790008545
|
||||
872a1e67-873a-4c49-a598-b38a670afcad,47187,0,0.33440664410591125,0.664151668548584,0.001441665575839579,incomplete,0,0,128000,47187.0,1.7980830743908882,4.785459488630295,31.987749971449375
|
||||
872a1e67-873a-4c49-a598-b38a670afcad,49500,1,0.46493276953697205,0.5347657203674316,0.0003015460097230971,incomplete,0,0,128000,49500.0,2.1763751283288,5.169540643692017,29.42441590130329
|
||||
872a1e67-873a-4c49-a598-b38a670afcad,52326,0,0.31294915080070496,0.6869400143623352,0.00011081220145570114,incomplete,0,0,128000,52326.0,2.1302076056599617,4.98233363032341,30.02820909023285
|
||||
872a1e67-873a-4c49-a598-b38a670afcad,54318,1,0.7077078223228455,0.2920091450214386,0.00028306824970059097,complete,1,1,128000,54318.0,1.9198330119252205,4.737791605293751,30.707416124641895
|
||||
a266b9ef-5e47-4434-a04c-ce4eacd52c19,18027,1,0.19850707054138184,0.8014131784439087,7.97970496932976e-05,incomplete,0,0,128000,18027.0,0.5511250346899033,4.951875656843185,30.43895773589611
|
||||
a266b9ef-5e47-4434-a04c-ce4eacd52c19,24768,1,0.45438897609710693,0.5453333258628845,0.00027762504760175943,incomplete,0,0,128000,24768.0,0.8649574592709541,5.036665126681328,29.373792000114918
|
||||
a266b9ef-5e47-4434-a04c-ce4eacd52c19,41219,0,0.20029887557029724,0.798045814037323,0.001655340543948114,incomplete,0,0,128000,41219.0,1.262083649635315,4.710624925792217,30.250207521021366
|
||||
a266b9ef-5e47-4434-a04c-ce4eacd52c19,51080,0,0.6161020994186401,0.3807879388332367,0.003109981305897236,complete,1,1,128000,51080.0,1.7816657200455666,4.898290149867535,38.34949992597103
|
||||
a266b9ef-5e47-4434-a04c-ce4eacd52c19,68313,0,0.8574621677398682,0.1419398933649063,0.0005979320267215371,complete,1,1,128000,68313.0,1.8581673502922058,5.039874464273453,29.415292665362358
|
||||
cb3238dc-3946-42a5-aad8-9b9a54226363,13540,1,0.6023414134979248,0.39736831188201904,0.0002902977867051959,complete,1,1,128000,13540.0,0.6275000050663948,5.3015416488051414,31.05229139328003
|
||||
cb3238dc-3946-42a5-aad8-9b9a54226363,19620,1,0.3794628083705902,0.6204359531402588,0.00010125405970029533,incomplete,0,0,128000,19620.0,0.6028758361935616,5.842084065079689,35.181958228349686
|
||||
cb3238dc-3946-42a5-aad8-9b9a54226363,27520,1,0.5865507125854492,0.41297394037246704,0.0004753079265356064,complete,1,1,128000,27520.0,0.7745418697595596,4.71708457916975,30.025207437574863
|
||||
cb3238dc-3946-42a5-aad8-9b9a54226363,41207,0,0.44317537546157837,0.5567837953567505,4.084650208824314e-05,incomplete,0,0,128000,41207.0,1.503458246588707,4.907291382551193,31.758290715515614
|
||||
cb3238dc-3946-42a5-aad8-9b9a54226363,49204,0,0.4868903160095215,0.513083815574646,2.5880843168124557e-05,incomplete,0,0,128000,49204.0,1.718541607260704,4.778958857059479,30.595874413847923
|
||||
cb3238dc-3946-42a5-aad8-9b9a54226363,58424,0,0.13133850693702698,0.8686069250106812,5.45754301128909e-05,incomplete,0,0,128000,58424.0,1.8835840746760368,4.6873753890395164,29.813292436301708
|
||||
cb3238dc-3946-42a5-aad8-9b9a54226363,60790,0,0.1166829988360405,0.8832864761352539,3.057053982047364e-05,incomplete,0,0,128000,60790.0,2.817000262439251,4.758957773447037,29.99345865100622
|
||||
cb3238dc-3946-42a5-aad8-9b9a54226363,70745,0,0.22430983185768127,0.7756540179252625,3.617277616285719e-05,incomplete,0,0,128000,70745.0,2.755166031420231,5.143124610185623,32.967125065624714
|
||||
cb3238dc-3946-42a5-aad8-9b9a54226363,73640,0,0.46023279428482056,0.5397213697433472,4.5886350562796e-05,incomplete,0,0,128000,73640.0,3.018917515873909,5.13325072824955,31.059250235557556
|
||||
e304f740-85d4-427e-9fd6-c5474e9cd737,11531,1,0.4405680298805237,0.5592595338821411,0.0001724747708067298,incomplete,0,0,128000,11531.0,0.40116626769304276,4.7815414145588875,31.708291731774807
|
||||
e304f740-85d4-427e-9fd6-c5474e9cd737,18842,1,0.6414244771003723,0.35465940833091736,0.003916137386113405,complete,1,1,128000,18842.0,0.5581667646765709,4.73874993622303,29.53079156577587
|
||||
e304f740-85d4-427e-9fd6-c5474e9cd737,29619,1,0.5816889405250549,0.41641655564308167,0.0018945817137137055,complete,1,1,128000,29619.0,0.842750072479248,4.706874489784241,29.850250110030174
|
||||
e304f740-85d4-427e-9fd6-c5474e9cd737,40701,0,0.3338589072227478,0.6659473180770874,0.0001937897759489715,incomplete,0,0,128000,40701.0,1.445750705897808,5.079125054180622,31.952625140547752
|
||||
e304f740-85d4-427e-9fd6-c5474e9cd737,54600,1,0.8655244708061218,0.13437898457050323,9.655235044192523e-05,complete,1,1,128000,54600.0,2.032499760389328,4.749624989926815,31.001625582575798
|
||||
e304f740-85d4-427e-9fd6-c5474e9cd737,62828,0,0.4587578773498535,0.5411435961723328,9.847679757513106e-05,incomplete,0,0,128000,62828.0,2.3750411346554756,4.857249557971954,32.79420919716358
|
||||
e304f740-85d4-427e-9fd6-c5474e9cd737,65819,0,0.08307088166475296,0.9169193506240845,9.722626600705553e-06,incomplete,0,0,128000,65819.0,2.238958142697811,4.7214580699801445,32.456833869218826
|
||||
|
@@ -0,0 +1,30 @@
|
||||
WITNESS CPU seed=42 [2.4581515390309505e-05, 7.737393025308847e-05, 0.9998980760574341] repeat identical
|
||||
2c83253a: 100/489 scores
|
||||
2c83253a: 200/489 scores
|
||||
2c83253a: 300/489 scores
|
||||
2c83253a: 400/489 scores
|
||||
2c83253a DONE: 489 scores, 6 checkpoints; repeat/future-suffix checks passed
|
||||
40ae686c: 100/441 scores
|
||||
40ae686c: 200/441 scores
|
||||
40ae686c: 300/441 scores
|
||||
40ae686c: 400/441 scores
|
||||
40ae686c DONE: 441 scores, 8 checkpoints; repeat/future-suffix checks passed
|
||||
872a1e67: 100/379 scores
|
||||
872a1e67: 200/379 scores
|
||||
872a1e67: 300/379 scores
|
||||
872a1e67 DONE: 379 scores, 9 checkpoints; repeat/future-suffix checks passed
|
||||
a266b9ef: 100/421 scores
|
||||
a266b9ef: 200/421 scores
|
||||
a266b9ef: 300/421 scores
|
||||
a266b9ef: 400/421 scores
|
||||
a266b9ef DONE: 421 scores, 5 checkpoints; repeat/future-suffix checks passed
|
||||
cb3238dc: 100/478 scores
|
||||
cb3238dc: 200/478 scores
|
||||
cb3238dc: 300/478 scores
|
||||
cb3238dc: 400/478 scores
|
||||
cb3238dc DONE: 478 scores, 9 checkpoints; repeat/future-suffix checks passed
|
||||
e304f740: 100/365 scores
|
||||
e304f740: 200/365 scores
|
||||
e304f740: 300/365 scores
|
||||
e304f740 DONE: 365 scores, 7 checkpoints; repeat/future-suffix checks passed
|
||||
{"native_argmax": {"n": 44, "tn": 17, "fp": 5, "fn": 9, "tp": 13, "accuracy": 0.6818181818181818, "false_reply_rate": 0.22727272727272727, "false_hold_rate": 0.4090909090909091}, "threshold_05": {"n": 44, "tn": 17, "fp": 5, "fn": 9, "tp": 13, "accuracy": 0.6818181818181818, "false_reply_rate": 0.22727272727272727, "false_hold_rate": 0.4090909090909091}, "class_counts": {"complete": 18, "incomplete": 26}}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,2 @@
|
||||
WITNESS CPU seed=42 [2.4581515390309505e-05, 7.737393025308847e-05, 0.9998980760574341] repeat identical
|
||||
{"load_ms": 576.1605408042669, "provenance": {"machine": "macOS-26.4.1-arm64-arm-64bit", "hardware": "Apple M4 Mac mini CPU / 16 GB", "python": "3.11.15 (main, Mar 25 2026, 02:58:47) [Clang 22.1.1 ]", "dependencies": {"numpy": "2.2.6", "onnxruntime": "1.24.4", "librosa": "0.11.0", "kaldi-native-fbank": "1.22.3", "soundfile": "0.13.1"}, "providers": ["CPUExecutionProvider"], "intra_op_threads": 1, "inter_op_threads": 1, "model_manifest": {"repo": "brgroup/TurnSense", "revision": "341dd68c9df87d74d35b9cee63a1dcf0fff24966", "version": "TurnSense1.0 FP32", "code_repo": "Bairong-Xdynamics/TurnSense", "code_revision": "c6d83e5b0d1355125b05eaf1b4b5c6c1acabb338", "files": {"model_fp32.onnx": "82256247dd38f0d9f2f213beec7ede1507347f84422fa48f29d565b613d89976", "am.mvn": "29b3c740a2c0cfc6b308126d31d7f265fa2be74f3bb095cd2f143ea970896ae5"}, "verified_against": "Hugging Face fixed revision LFS SHA256 / Git blob SHA1"}, "script_sha256": "23215a6db3c39f6819761adf1e6cd46a8f7c0c7767d1a3640bc928ecd548b2fb", "annotation_sha256": "780e50f976daf039ab976ea155bdb5d1ca5fffc792f9db7adf8063ec3048ff8e", "frontend_sha256": "22808c254ea2150e290e6bdcf115322a723b67fc4e53b1ef02b8f3f42697187d", "official_infer_sha256": "40295fb42a36ae4ecca30093d6bc37acbca9c75304b1a722dba73fd3b30498df"}}
|
||||
@@ -0,0 +1,76 @@
|
||||
{
|
||||
"model": "TurnSense 1.0 FP32",
|
||||
"native_argmax": {
|
||||
"n": 44,
|
||||
"tn": 17,
|
||||
"fp": 5,
|
||||
"fn": 9,
|
||||
"tp": 13,
|
||||
"accuracy": 0.6818181818181818,
|
||||
"false_reply_rate": 0.22727272727272727,
|
||||
"false_hold_rate": 0.4090909090909091
|
||||
},
|
||||
"threshold_05": {
|
||||
"n": 44,
|
||||
"tn": 17,
|
||||
"fp": 5,
|
||||
"fn": 9,
|
||||
"tp": 13,
|
||||
"accuracy": 0.6818181818181818,
|
||||
"false_reply_rate": 0.22727272727272727,
|
||||
"false_hold_rate": 0.4090909090909091
|
||||
},
|
||||
"class_counts": {
|
||||
"complete": 18,
|
||||
"incomplete": 26
|
||||
},
|
||||
"load_ms": 657.7007500454783,
|
||||
"method": "TurnSense1.0 FP32, official code/frontend/CMVN and 3-class postprocessing. CPU one intra/inter-op thread. User channel 0 int16/32768, crop received prefix BEFORE librosa resample to 16kHz, last 8s, no future ASR/audio or silence/timeout gates. Score=P(complete), native argmax complete=>1 else0; separately report threshold0.5. Full-prefix resampling is offline causal-window evaluation, not an incremental frontend benchmark.",
|
||||
"provenance": {
|
||||
"machine": "macOS-26.4.1-arm64-arm-64bit",
|
||||
"hardware": "Apple M4 Mac mini CPU / 16 GB",
|
||||
"python": "3.11.15 (main, Mar 25 2026, 02:58:47) [Clang 22.1.1 ]",
|
||||
"dependencies": {
|
||||
"numpy": "2.2.6",
|
||||
"onnxruntime": "1.24.4",
|
||||
"librosa": "0.11.0",
|
||||
"kaldi-native-fbank": "1.22.3",
|
||||
"soundfile": "0.13.1"
|
||||
},
|
||||
"providers": [
|
||||
"CPUExecutionProvider"
|
||||
],
|
||||
"intra_op_threads": 1,
|
||||
"inter_op_threads": 1,
|
||||
"model_manifest": {
|
||||
"repo": "brgroup/TurnSense",
|
||||
"revision": "341dd68c9df87d74d35b9cee63a1dcf0fff24966",
|
||||
"version": "TurnSense1.0 FP32",
|
||||
"code_repo": "Bairong-Xdynamics/TurnSense",
|
||||
"code_revision": "c6d83e5b0d1355125b05eaf1b4b5c6c1acabb338",
|
||||
"files": {
|
||||
"model_fp32.onnx": "82256247dd38f0d9f2f213beec7ede1507347f84422fa48f29d565b613d89976",
|
||||
"am.mvn": "29b3c740a2c0cfc6b308126d31d7f265fa2be74f3bb095cd2f143ea970896ae5"
|
||||
},
|
||||
"verified_against": "Hugging Face fixed revision LFS SHA256 / Git blob SHA1"
|
||||
},
|
||||
"script_sha256": "23215a6db3c39f6819761adf1e6cd46a8f7c0c7767d1a3640bc928ecd548b2fb",
|
||||
"annotation_sha256": "780e50f976daf039ab976ea155bdb5d1ca5fffc792f9db7adf8063ec3048ff8e",
|
||||
"frontend_sha256": "22808c254ea2150e290e6bdcf115322a723b67fc4e53b1ef02b8f3f42697187d",
|
||||
"official_infer_sha256": "40295fb42a36ae4ecca30093d6bc37acbca9c75304b1a722dba73fd3b30498df"
|
||||
},
|
||||
"latency_ms": {
|
||||
"preparation_ms": {
|
||||
"p50": 1.7886869609355927,
|
||||
"p95": 3.0542627442628145
|
||||
},
|
||||
"frontend_ms": {
|
||||
"p50": 4.848978947848082,
|
||||
"p95": 5.63460667617619
|
||||
},
|
||||
"inference_ms": {
|
||||
"p50": 31.06177132576704,
|
||||
"p95": 35.107012651860714
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
# TurnSense 1.0 FP32 韩泰评估
|
||||
|
||||
本机 Apple M4 Mac mini、CPU；六通录音，当前 44 个判断点，正负各 22。只使用判断时已收到的用户左声道音频、末尾最多 8 秒，使用官方特征前端与 CMVN，未输入标注转写。
|
||||
|
||||
| 判断规则 | 正确 / 总数 | 准确率 | 误接话（应等待却接话） | 漏接话（应接话却等待） |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 原生 argmax：complete=>1，其余=>0 | 30 / 44 | 68.18% | 5 / 22（22.73%） | 9 / 22（40.91%） |
|
||||
| P(complete) ≥ 0.5 | 30 / 44 | 68.18% | 5 / 22（22.73%） | 9 / 22（40.91%） |
|
||||
|
||||
本批两种规则得到相同二分类结果，但它们在一般情况下并不等价。原生类别：complete 18 点、incomplete 26 点、invalid 0 点。invalid 输出仍保留，不能据此评估噪声识别能力，因为本批没有独立 invalid 真值。
|
||||
|
||||
## 计算耗时
|
||||
|
||||
| 阶段 | p50 | p95 |
|
||||
| --- | --- | --- |
|
||||
| 裁取/前缀重采样 | 1.79 ms | 3.05 ms |
|
||||
| 官方特征前端 | 4.85 ms | 5.63 ms |
|
||||
| ONNX 模型推理 | 31.06 ms | 35.11 ms |
|
||||
|
||||
模型构造 657.70 ms，不含 Python 启动/import，不能称为完整冷启动。CPU intra/inter-op 各 1 线程，已预热；前缀重采样和特征重算不等于在线增量开销。阶段分位数不应相加成总耗时分位数。
|
||||
|
||||
## 解读与比较边界
|
||||
|
||||
同一当前标注哈希的 LiveKit v1-mini 在 0.5 下为 29/44，误接话 11、漏接话 4；TurnSense 在 0.5 下为 30/44，误接话 5、漏接话 9。可见这批点上更倾向等待：少误接话，但更多应接话时未接话。样本小、上下文长度不同（8 秒 vs 1.2 秒），不能把这 1 点正确率差异当成稳定优势，更不能宣布已解决抢话。
|
||||
|
||||
当前标注第二通一个点已由旧 VAP / Smart Turn 快照的 48348 ms 改为 48536 ms，不能直接用其旧 44 点汇总严格排名。原曲线空心点保留各自评分时刻。
|
||||
|
||||
## 可核验性
|
||||
|
||||
- 全部判断点重复模型推理结果完全一致。
|
||||
- 将判断点之后的音频置零，准备出的模型输入不变。
|
||||
- 权重 SHA256 与固定 Hugging Face 修订的 LFS SHA256 一致；CMVN 用 Git blob SHA1 核对，并另记 SHA256。
|
||||
- 模型、CMVN、脚本、前端、标注、录音哈希与环境版本均写入结果，独立环境文档已由另一代理按说明复现 smoke。
|
||||
- 连续曲线为每 200 ms 实际推理及每个判断点的精确分数，共 2573 个采样点，非插值生成。
|
||||
|
||||
44 个手选判断点仅为探索性评估；点级误接话率不等于整段对话抢话率，用户声道可能有机器人回声。不加 VAD 门槛、静音延迟或超时策略。模型为 1.0 FP32，本次没有测试 INT8 或 1.1。
|
||||
Reference in New Issue
Block a user