Archive EasyCom sparse supervision results
This commit is contained in:
@@ -1241,7 +1241,9 @@ sparse LoRA recipe:
|
||||
| Checkpoint | Full-timeline mean mAP | Strict causal mean mAP |
|
||||
|---|---:|---:|
|
||||
| Frozen Thinker dense-cache control | 58.16091 | 58.17146 |
|
||||
| Frozen states + sparse supervision, 50 epochs | 57.20379 | 57.21367 |
|
||||
| Sparse LoRA epoch1 | 56.52480 | 56.53450 |
|
||||
| Frozen-head initialized LoRA, 3 epochs | 56.19608 | 56.20535 |
|
||||
| Sparse LoRA weights-only epoch2 | 54.19850 | 54.20711 |
|
||||
|
||||
Epoch2 is a weights-only continuation: adapter/head weights resume, but AdamW
|
||||
@@ -1251,6 +1253,14 @@ regression is camera-wearer speaking. Do not continue the same recipe to a
|
||||
third run. The complete snapshot and per-class metrics are archived in
|
||||
[`results/easycom_omni_lora_sparse_epoch2_2026-08-24/`](results/easycom_omni_lora_sparse_epoch2_2026-08-24/).
|
||||
|
||||
The frozen-state sparse-supervision and frozen-head initialized three-epoch
|
||||
LoRA tables are archived in
|
||||
[`results/easycom_sparse_supervision_vs_lora_2026-08-24/`](results/easycom_sparse_supervision_vs_lora_2026-08-24/README.md).
|
||||
At 0.2 seconds, sparse supervision alone changes mAP from `71.69695` to
|
||||
`71.69194` (effectively unchanged), while the three-epoch LoRA result is
|
||||
`71.08589`. This isolates the short-horizon regression primarily to Thinker
|
||||
updating rather than sparse target selection.
|
||||
|
||||
For a speed-only alternative, `train_omni_turn_lora.py` supports a two-GPU
|
||||
producer-consumer benchmark. The producer GPU runs the frozen AV towers and
|
||||
places current-tail embeddings in a bounded CPU queue while the consumer GPU
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
# EasyCom open-stream context: 8 seconds versus 20 seconds
|
||||
|
||||
Recorded on **2026-08-23**. This is a single-run experiment snapshot, not a
|
||||
final multi-seed paper result.
|
||||
|
||||
## Experiment contract
|
||||
|
||||
- Dataset: EasyCom with EgoSpeak's official split.
|
||||
- Train sessions: 7, 11, 10, 6, 9, 4, 8, 5, and 12.
|
||||
- Test sessions: 3, 2, and 1.
|
||||
- Representation: frozen Qwen2.5-Omni-3B Thinker, open-stream replay.
|
||||
- Head: MLP `2048 -> 1024 -> 30`, reshaped to `10 x 3`.
|
||||
- Video sampling: 2 FPS.
|
||||
- Prediction rate: 5 Hz.
|
||||
- Context budgets: 8 seconds and 20 seconds.
|
||||
- Targets: ten future horizons from 0.2 through 2.0 seconds.
|
||||
- Loss/evaluation: EgoSpeak LSTR MCE and Table 2 frame AP/mAP protocol.
|
||||
|
||||
The heads were trained separately for their corresponding representations. A
|
||||
head trained on the 20-second states was not reused on the 8-second states.
|
||||
|
||||
## Summary
|
||||
|
||||
All values are percentages.
|
||||
|
||||
| Protocol | 8 seconds | 20 seconds | 8s minus 20s |
|
||||
|---|---:|---:|---:|
|
||||
| EgoSpeak full timeline | **58.51937** | 58.316 | **+0.20337** |
|
||||
| Strict causal valid only | **58.53015** | 58.326 | **+0.20415** |
|
||||
|
||||
The full-timeline per-horizon comparison is:
|
||||
|
||||
| Horizon | Time | 8s mAP | 20s mAP | 8s minus 20s |
|
||||
|---:|---:|---:|---:|---:|
|
||||
| 1 | 0.2s | **71.99768** | 69.756 | **+2.24168** |
|
||||
| 2 | 0.4s | **66.74289** | 65.163 | **+1.57989** |
|
||||
| 3 | 0.6s | **62.38972** | 61.578 | **+0.81172** |
|
||||
| 4 | 0.8s | **59.12229** | 58.783 | **+0.33929** |
|
||||
| 5 | 1.0s | 56.71380 | **56.913** | -0.19920 |
|
||||
| 6 | 1.2s | 55.36300 | **55.727** | -0.36400 |
|
||||
| 7 | 1.4s | 54.21297 | **54.819** | -0.60603 |
|
||||
| 8 | 1.6s | 53.43221 | **53.957** | -0.52479 |
|
||||
| 9 | 1.8s | 52.84703 | **53.459** | -0.61197 |
|
||||
| 10 | 2.0s | 52.37207 | **53.002** | -0.62993 |
|
||||
| Mean | — | **58.51937** | 58.316 | **+0.20337** |
|
||||
|
||||
For the full-timeline protocol, averaging AP over the ten horizons gives:
|
||||
|
||||
| Class | 8s mean AP | 20s mean AP | 8s minus 20s |
|
||||
|---|---:|---:|---:|
|
||||
| Background | 39.63094 | **40.49350** | -0.86256 |
|
||||
| Camera wearer speaking | **58.91206** | 58.59400 | +0.31806 |
|
||||
| Other people speaking | **77.01510** | 75.85970 | +1.15540 |
|
||||
|
||||
## Interpretation
|
||||
|
||||
The aggregate 8-second advantage is only 0.20 mAP, but it is not distributed
|
||||
uniformly. The 8-second representation is substantially stronger at 0.2--0.6
|
||||
seconds, while the 20-second representation becomes stronger from 1.0 seconds
|
||||
onward. This looks like a short/long-horizon tradeoff rather than evidence that
|
||||
one context budget is uniformly better.
|
||||
|
||||
The class pattern is also mixed: 8 seconds improves the two speaking classes,
|
||||
especially other-person speech, but reduces background AP. Because the heads
|
||||
were trained once and the aggregate difference is small, training variance
|
||||
could plausibly reverse the mean comparison. Identical multi-seed runs are
|
||||
required before selecting the context budget for the paper.
|
||||
|
||||
## Files and provenance
|
||||
|
||||
- `raw_8s_console.txt` preserves the reported 8-second console output.
|
||||
- `metrics.csv` contains both contexts, protocols, horizons, and classes.
|
||||
- `summary.json` contains the machine-readable aggregate comparison.
|
||||
- The 20-second values come from the previously archived
|
||||
`easycom_open_stream_vs_causal_prefix_2026-08-19` snapshot and therefore
|
||||
have only three-decimal precision.
|
||||
- Aggregate differences use the separately reported aggregate means. Averaging
|
||||
differences reconstructed from the rounded 20-second horizon rows can vary
|
||||
slightly from those aggregate differences.
|
||||
- The original remote `metrics.json` files have not yet been copied here.
|
||||
@@ -0,0 +1,41 @@
|
||||
context_seconds,protocol,horizon,seconds,background_ap_percent,camera_wearer_ap_percent,other_people_ap_percent,map_percent,num_steps
|
||||
8,egospeak_full_timeline,1,0.2,54.84736,76.45639,84.68928,71.99768,23048
|
||||
8,egospeak_full_timeline,2,0.4,47.46373,71.02340,81.74154,66.74289,23048
|
||||
8,egospeak_full_timeline,3,0.6,42.19659,65.57303,79.39953,62.38972,23048
|
||||
8,egospeak_full_timeline,4,0.8,38.81557,60.96834,77.58298,59.12229,23048
|
||||
8,egospeak_full_timeline,5,1.0,36.79821,57.21128,76.13193,56.71380,23048
|
||||
8,egospeak_full_timeline,6,1.2,36.09944,54.87811,75.11144,55.36300,23048
|
||||
8,egospeak_full_timeline,7,1.4,35.41214,52.71169,74.51507,54.21297,23048
|
||||
8,egospeak_full_timeline,8,1.6,35.15599,51.16932,73.97133,53.43221,23048
|
||||
8,egospeak_full_timeline,9,1.8,34.85984,50.01901,73.66226,52.84703,23048
|
||||
8,egospeak_full_timeline,10,2.0,34.66053,49.11007,73.34560,52.37207,23048
|
||||
8,causal_valid_only,1,0.2,54.84736,76.45639,84.69504,71.99960,23045
|
||||
8,causal_valid_only,2,0.4,47.46373,71.02340,81.75175,66.74630,23042
|
||||
8,causal_valid_only,3,0.6,42.19659,65.57933,79.41330,62.39641,23039
|
||||
8,causal_valid_only,4,0.8,38.81557,60.97945,77.59974,59.13158,23036
|
||||
8,causal_valid_only,5,1.0,36.79821,57.22612,76.15127,56.72520,23033
|
||||
8,causal_valid_only,6,1.2,36.09944,54.89639,75.13207,55.37597,23030
|
||||
8,causal_valid_only,7,1.4,35.41214,52.73278,74.53730,54.22741,23027
|
||||
8,causal_valid_only,8,1.6,35.15831,51.18915,73.99506,53.44751,23024
|
||||
8,causal_valid_only,9,1.8,34.86210,50.03791,73.68883,52.86295,23021
|
||||
8,causal_valid_only,10,2.0,34.66276,49.12823,73.37487,52.38862,23018
|
||||
20,egospeak_full_timeline,1,0.2,53.605,74.331,81.331,69.756,23048
|
||||
20,egospeak_full_timeline,2,0.4,47.066,69.402,79.020,65.163,23048
|
||||
20,egospeak_full_timeline,3,0.6,42.615,64.600,77.520,61.578,23048
|
||||
20,egospeak_full_timeline,4,0.8,39.463,60.641,76.244,58.783,23048
|
||||
20,egospeak_full_timeline,5,1.0,38.002,57.371,75.367,56.913,23048
|
||||
20,egospeak_full_timeline,6,1.2,37.398,55.258,74.523,55.727,23048
|
||||
20,egospeak_full_timeline,7,1.4,36.933,53.400,74.126,54.819,23048
|
||||
20,egospeak_full_timeline,8,1.6,36.653,51.502,73.716,53.957,23048
|
||||
20,egospeak_full_timeline,9,1.8,36.527,50.346,73.504,53.459,23048
|
||||
20,egospeak_full_timeline,10,2.0,36.673,49.089,73.246,53.002,23048
|
||||
20,causal_valid_only,1,0.2,53.605,74.331,81.336,69.757,23045
|
||||
20,causal_valid_only,2,0.4,47.066,69.402,79.029,65.166,23042
|
||||
20,causal_valid_only,3,0.6,42.615,64.607,77.532,61.585,23039
|
||||
20,causal_valid_only,4,0.8,39.463,60.652,76.259,58.792,23036
|
||||
20,causal_valid_only,5,1.0,38.002,57.386,75.385,56.924,23033
|
||||
20,causal_valid_only,6,1.2,37.398,55.277,74.543,55.739,23030
|
||||
20,causal_valid_only,7,1.4,36.933,53.421,74.148,54.834,23027
|
||||
20,causal_valid_only,8,1.6,36.656,51.522,73.740,53.973,23024
|
||||
20,causal_valid_only,9,1.8,36.529,50.365,73.530,53.475,23021
|
||||
20,causal_valid_only,10,2.0,36.676,49.107,73.275,53.019,23018
|
||||
|
@@ -0,0 +1,45 @@
|
||||
egospeak_full_timeline:
|
||||
Action anticipation (0.20s) perframe mAP: 71.99768
|
||||
AP: background=54.84736, camera_wearer_speaking=76.45639, other_people_speaking=84.68928; n=23048
|
||||
Action anticipation (0.40s) perframe mAP: 66.74289
|
||||
AP: background=47.46373, camera_wearer_speaking=71.02340, other_people_speaking=81.74154; n=23048
|
||||
Action anticipation (0.60s) perframe mAP: 62.38972
|
||||
AP: background=42.19659, camera_wearer_speaking=65.57303, other_people_speaking=79.39953; n=23048
|
||||
Action anticipation (0.80s) perframe mAP: 59.12229
|
||||
AP: background=38.81557, camera_wearer_speaking=60.96834, other_people_speaking=77.58298; n=23048
|
||||
Action anticipation (1.00s) perframe mAP: 56.71380
|
||||
AP: background=36.79821, camera_wearer_speaking=57.21128, other_people_speaking=76.13193; n=23048
|
||||
Action anticipation (1.20s) perframe mAP: 55.36300
|
||||
AP: background=36.09944, camera_wearer_speaking=54.87811, other_people_speaking=75.11144; n=23048
|
||||
Action anticipation (1.40s) perframe mAP: 54.21297
|
||||
AP: background=35.41214, camera_wearer_speaking=52.71169, other_people_speaking=74.51507; n=23048
|
||||
Action anticipation (1.60s) perframe mAP: 53.43221
|
||||
AP: background=35.15599, camera_wearer_speaking=51.16932, other_people_speaking=73.97133; n=23048
|
||||
Action anticipation (1.80s) perframe mAP: 52.84703
|
||||
AP: background=34.85984, camera_wearer_speaking=50.01901, other_people_speaking=73.66226; n=23048
|
||||
Action anticipation (2.00s) perframe mAP: 52.37207
|
||||
AP: background=34.66053, camera_wearer_speaking=49.11007, other_people_speaking=73.34560; n=23048
|
||||
Action anticipation (mean) perframe mAP: 58.51937
|
||||
|
||||
causal_valid_only:
|
||||
Action anticipation (0.20s) perframe mAP: 71.99960
|
||||
AP: background=54.84736, camera_wearer_speaking=76.45639, other_people_speaking=84.69504; n=23045
|
||||
Action anticipation (0.40s) perframe mAP: 66.74630
|
||||
AP: background=47.46373, camera_wearer_speaking=71.02340, other_people_speaking=81.75175; n=23042
|
||||
Action anticipation (0.60s) perframe mAP: 62.39641
|
||||
AP: background=42.19659, camera_wearer_speaking=65.57933, other_people_speaking=79.41330; n=23039
|
||||
Action anticipation (0.80s) perframe mAP: 59.13158
|
||||
AP: background=38.81557, camera_wearer_speaking=60.97945, other_people_speaking=77.59974; n=23036
|
||||
Action anticipation (1.00s) perframe mAP: 56.72520
|
||||
AP: background=36.79821, camera_wearer_speaking=57.22612, other_people_speaking=76.15127; n=23033
|
||||
Action anticipation (1.20s) perframe mAP: 55.37597
|
||||
AP: background=36.09944, camera_wearer_speaking=54.89639, other_people_speaking=75.13207; n=23030
|
||||
Action anticipation (1.40s) perframe mAP: 54.22741
|
||||
AP: background=35.41214, camera_wearer_speaking=52.73278, other_people_speaking=74.53730; n=23027
|
||||
Action anticipation (1.60s) perframe mAP: 53.44751
|
||||
AP: background=35.15831, camera_wearer_speaking=51.18915, other_people_speaking=73.99506; n=23024
|
||||
Action anticipation (1.80s) perframe mAP: 52.86295
|
||||
AP: background=34.86210, camera_wearer_speaking=50.03791, other_people_speaking=73.68883; n=23021
|
||||
Action anticipation (2.00s) perframe mAP: 52.38862
|
||||
AP: background=34.66276, camera_wearer_speaking=49.12823, other_people_speaking=73.37487; n=23018
|
||||
Action anticipation (mean) perframe mAP: 58.53015
|
||||
@@ -0,0 +1,76 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"recorded_date": "2026-08-23",
|
||||
"status": "single_run_snapshot",
|
||||
"dataset": "EasyCom",
|
||||
"representation": "Qwen2.5-Omni-3B open-stream replay",
|
||||
"video_fps": 2,
|
||||
"prediction_step_seconds": 0.2,
|
||||
"split": {
|
||||
"train_sessions": [7, 11, 10, 6, 9, 4, 8, 5, 12],
|
||||
"test_sessions": [3, 2, 1]
|
||||
},
|
||||
"metrics_percent": {
|
||||
"8_seconds": {
|
||||
"egospeak_full_timeline_mean_map": 58.51937,
|
||||
"causal_valid_only_mean_map": 58.53015,
|
||||
"egospeak_full_timeline_horizon_map": [
|
||||
71.99768,
|
||||
66.74289,
|
||||
62.38972,
|
||||
59.12229,
|
||||
56.71380,
|
||||
55.36300,
|
||||
54.21297,
|
||||
53.43221,
|
||||
52.84703,
|
||||
52.37207
|
||||
]
|
||||
},
|
||||
"20_seconds": {
|
||||
"egospeak_full_timeline_mean_map": 58.316,
|
||||
"causal_valid_only_mean_map": 58.326,
|
||||
"egospeak_full_timeline_horizon_map": [
|
||||
69.756,
|
||||
65.163,
|
||||
61.578,
|
||||
58.783,
|
||||
56.913,
|
||||
55.727,
|
||||
54.819,
|
||||
53.957,
|
||||
53.459,
|
||||
53.002
|
||||
]
|
||||
},
|
||||
"8_seconds_minus_20_seconds": {
|
||||
"egospeak_full_timeline_mean_map": 0.20337,
|
||||
"causal_valid_only_mean_map": 0.20415,
|
||||
"egospeak_full_timeline_horizon_map": [
|
||||
2.24168,
|
||||
1.57989,
|
||||
0.81172,
|
||||
0.33929,
|
||||
-0.19920,
|
||||
-0.36400,
|
||||
-0.60603,
|
||||
-0.52479,
|
||||
-0.61197,
|
||||
-0.62993
|
||||
],
|
||||
"egospeak_full_timeline_mean_class_ap": {
|
||||
"background": -0.86256,
|
||||
"camera_wearer_speaking": 0.31806,
|
||||
"other_people_speaking": 1.15540
|
||||
}
|
||||
}
|
||||
},
|
||||
"provenance": {
|
||||
"8_seconds": "User-reported infer_turn_head.py console output; preserved in raw_8s_console.txt",
|
||||
"20_seconds": "Previously archived 2026-08-19 open-stream snapshot; values available to three decimals",
|
||||
"precision_note": "Aggregate differences use the separately reported aggregate means. Per-horizon and per-class differences use the available row values and can differ slightly when averaged because the archived 20-second rows were rounded to three decimals.",
|
||||
"raw_8_seconds_metrics_json_archived_here": false,
|
||||
"raw_20_seconds_metrics_json_archived_here": false,
|
||||
"multi_seed": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
# EasyCom: sparse supervision versus Thinker LoRA
|
||||
|
||||
Recorded on **2026-08-24** from user-reported remote evaluation tables. These
|
||||
are single-run diagnostic results, not final multi-seed paper numbers.
|
||||
|
||||
## Experiment contract
|
||||
|
||||
- Dataset and split: EasyCom with EgoSpeak's official nine training sessions
|
||||
and test Sessions 3, 2, and 1.
|
||||
- Prediction rate: 5 Hz; ten future horizons from 0.2 to 2.0 seconds.
|
||||
- Video sampling: 2 FPS.
|
||||
- Evaluator: EgoSpeak full-timeline AP/mAP plus strict causal-valid AP/mAP.
|
||||
- Sparse policy: ordinary anchors at 1 Hz plus 5 Hz anchors around annotated
|
||||
target transitions.
|
||||
- Evaluation is dense at every 5 Hz test timestamp.
|
||||
- Frozen sparse run: previously extracted frozen Thinker states; only the head
|
||||
is trained for 50 epochs on sparse target indices.
|
||||
- LoRA three-epoch run: initialized with the trained frozen head, then updated
|
||||
the Thinker LoRA and head for three epochs.
|
||||
|
||||
The remote `metrics.json`, exact checkpoint configuration, git revision, and
|
||||
feature-root path were not copied into this repository. The context/feature
|
||||
identity with the frozen 8-second control must therefore be verified before
|
||||
publication.
|
||||
|
||||
## Summary
|
||||
|
||||
All values are percentages.
|
||||
|
||||
| Experiment | 0.2s full mAP | Full mean mAP | Strict mean mAP |
|
||||
|---|---:|---:|---:|
|
||||
| Frozen Thinker dense-cache control | 71.69695 | 58.16091 | 58.17146 |
|
||||
| Frozen states + sparse supervision, 50 epochs | **71.69194** | **57.20379** | **57.21367** |
|
||||
| Sparse LoRA epoch 1 | 67.40167 | 56.52480 | 56.53450 |
|
||||
| Frozen-head initialized LoRA, 3 epochs | **71.08589** | **56.19608** | **56.20535** |
|
||||
| Sparse LoRA weights-only epoch 2 | 65.55214 | 54.19850 | 54.20711 |
|
||||
|
||||
The epoch-2 continuation is archived separately in
|
||||
[`../easycom_omni_lora_sparse_epoch2_2026-08-24/`](../easycom_omni_lora_sparse_epoch2_2026-08-24/README.md).
|
||||
|
||||
## Frozen states with sparse supervision
|
||||
|
||||
| Horizon | Full mAP | Strict mAP | Full background AP | Full wearer AP | Full others AP |
|
||||
|---:|---:|---:|---:|---:|---:|
|
||||
| 0.2s | 71.69194 | 71.69383 | 54.64681 | 76.09602 | 84.33299 |
|
||||
| 0.4s | 65.98617 | 65.98941 | 46.86427 | 70.48309 | 80.61116 |
|
||||
| 0.6s | 61.14072 | 61.14698 | 41.30997 | 64.40092 | 77.71127 |
|
||||
| 0.8s | 57.39854 | 57.40704 | 37.50173 | 59.07507 | 75.61882 |
|
||||
| 1.0s | 54.57513 | 54.58522 | 35.47209 | 54.41110 | 73.84220 |
|
||||
| 1.2s | 53.70320 | 53.71497 | 35.16647 | 52.41903 | 73.52409 |
|
||||
| 1.4s | 52.74547 | 52.75864 | 34.64200 | 50.53203 | 73.06238 |
|
||||
| 1.6s | 52.04826 | 52.06226 | 34.36442 | 49.18784 | 72.59253 |
|
||||
| 1.8s | 51.55154 | 51.56613 | 34.13135 | 48.17972 | 72.34356 |
|
||||
| 2.0s | 51.19693 | 51.21218 | 33.99937 | 47.47712 | 72.11430 |
|
||||
| Mean | **57.20379** | **57.21367** | — | — | — |
|
||||
|
||||
## Frozen-head initialized LoRA after three epochs
|
||||
|
||||
| Horizon | Full mAP | Strict mAP | Full background AP | Full wearer AP | Full others AP |
|
||||
|---:|---:|---:|---:|---:|---:|
|
||||
| 0.2s | 71.08589 | 71.08773 | 53.71399 | 75.95995 | 83.58373 |
|
||||
| 0.4s | 65.20613 | 65.20927 | 46.04420 | 69.65188 | 79.92232 |
|
||||
| 0.6s | 60.10484 | 60.11088 | 40.44134 | 62.71065 | 77.16253 |
|
||||
| 0.8s | 56.35250 | 56.36069 | 36.98112 | 56.61255 | 75.46383 |
|
||||
| 1.0s | 53.48271 | 53.49230 | 35.10974 | 51.63824 | 73.70016 |
|
||||
| 1.2s | 52.50271 | 52.51364 | 34.83634 | 49.81663 | 72.85516 |
|
||||
| 1.4s | 51.46987 | 51.48201 | 34.06639 | 48.01675 | 72.32646 |
|
||||
| 1.6s | 50.91134 | 50.92429 | 34.04663 | 46.85967 | 71.82772 |
|
||||
| 1.8s | 50.54008 | 50.55369 | 33.82003 | 46.15156 | 71.64864 |
|
||||
| 2.0s | 50.30475 | 50.31903 | 33.74069 | 45.80527 | 71.36829 |
|
||||
| Mean | **56.19608** | **56.20535** | — | — | — |
|
||||
|
||||
## 0.2-second diagnosis
|
||||
|
||||
Sparse supervision alone changes 0.2-second full-timeline mAP by only
|
||||
`-0.00501` relative to the frozen dense-cache control. The three-epoch LoRA
|
||||
run changes it by `-0.61106`. The much larger epoch-1 and weights-only epoch-2
|
||||
regressions show that short-horizon degradation is primarily associated with
|
||||
updating the Thinker representation, not with dropping ordinary supervision
|
||||
anchors.
|
||||
|
||||
The next required control is a 2-by-2 checkpoint evaluation: frozen Thinker
|
||||
with frozen head, frozen Thinker with LoRA-trained head, LoRA Thinker with
|
||||
frozen head, and LoRA Thinker with LoRA-trained head. Select LoRA checkpoints
|
||||
on a training-session development split using 0.2-second mAP rather than the
|
||||
official test sessions.
|
||||
|
||||
## Provenance
|
||||
|
||||
- Source: user-reported remote `infer_turn_head.py` and
|
||||
`eval_omni_turn_lora.py` console tables.
|
||||
- Raw remote metrics archived: no.
|
||||
- Multi-seed: no.
|
||||
- Publication status: diagnostic only.
|
||||
@@ -0,0 +1,41 @@
|
||||
experiment,protocol,horizon_seconds,map,background_ap,camera_wearer_speaking_ap,other_people_speaking_ap,num_steps
|
||||
frozen_states_sparse_supervision_50_epochs,egospeak_full_timeline,0.2,71.69194,54.64681,76.09602,84.33299,23048
|
||||
frozen_states_sparse_supervision_50_epochs,egospeak_full_timeline,0.4,65.98617,46.86427,70.48309,80.61116,23048
|
||||
frozen_states_sparse_supervision_50_epochs,egospeak_full_timeline,0.6,61.14072,41.30997,64.40092,77.71127,23048
|
||||
frozen_states_sparse_supervision_50_epochs,egospeak_full_timeline,0.8,57.39854,37.50173,59.07507,75.61882,23048
|
||||
frozen_states_sparse_supervision_50_epochs,egospeak_full_timeline,1.0,54.57513,35.47209,54.41110,73.84220,23048
|
||||
frozen_states_sparse_supervision_50_epochs,egospeak_full_timeline,1.2,53.70320,35.16647,52.41903,73.52409,23048
|
||||
frozen_states_sparse_supervision_50_epochs,egospeak_full_timeline,1.4,52.74547,34.64200,50.53203,73.06238,23048
|
||||
frozen_states_sparse_supervision_50_epochs,egospeak_full_timeline,1.6,52.04826,34.36442,49.18784,72.59253,23048
|
||||
frozen_states_sparse_supervision_50_epochs,egospeak_full_timeline,1.8,51.55154,34.13135,48.17972,72.34356,23048
|
||||
frozen_states_sparse_supervision_50_epochs,egospeak_full_timeline,2.0,51.19693,33.99937,47.47712,72.11430,23048
|
||||
frozen_states_sparse_supervision_50_epochs,causal_valid_only,0.2,71.69383,54.64681,76.09602,84.33868,23045
|
||||
frozen_states_sparse_supervision_50_epochs,causal_valid_only,0.4,65.98941,46.86427,70.48309,80.62088,23042
|
||||
frozen_states_sparse_supervision_50_epochs,causal_valid_only,0.6,61.14698,41.30997,64.40704,77.72392,23039
|
||||
frozen_states_sparse_supervision_50_epochs,causal_valid_only,0.8,57.40704,37.50173,59.08557,75.63383,23036
|
||||
frozen_states_sparse_supervision_50_epochs,causal_valid_only,1.0,54.58522,35.47209,54.42458,73.85900,23033
|
||||
frozen_states_sparse_supervision_50_epochs,causal_valid_only,1.2,53.71497,35.16647,52.43571,73.54273,23030
|
||||
frozen_states_sparse_supervision_50_epochs,causal_valid_only,1.4,52.75864,34.64200,50.55135,73.08256,23027
|
||||
frozen_states_sparse_supervision_50_epochs,causal_valid_only,1.6,52.06226,34.36659,49.20607,72.61411,23024
|
||||
frozen_states_sparse_supervision_50_epochs,causal_valid_only,1.8,51.56613,34.13348,48.19712,72.36779,23021
|
||||
frozen_states_sparse_supervision_50_epochs,causal_valid_only,2.0,51.21218,34.00148,47.49396,72.14111,23018
|
||||
frozen_head_initialized_lora_3_epochs,egospeak_full_timeline,0.2,71.08589,53.71399,75.95995,83.58373,23048
|
||||
frozen_head_initialized_lora_3_epochs,egospeak_full_timeline,0.4,65.20613,46.04420,69.65188,79.92232,23048
|
||||
frozen_head_initialized_lora_3_epochs,egospeak_full_timeline,0.6,60.10484,40.44134,62.71065,77.16253,23048
|
||||
frozen_head_initialized_lora_3_epochs,egospeak_full_timeline,0.8,56.35250,36.98112,56.61255,75.46383,23048
|
||||
frozen_head_initialized_lora_3_epochs,egospeak_full_timeline,1.0,53.48271,35.10974,51.63824,73.70016,23048
|
||||
frozen_head_initialized_lora_3_epochs,egospeak_full_timeline,1.2,52.50271,34.83634,49.81663,72.85516,23048
|
||||
frozen_head_initialized_lora_3_epochs,egospeak_full_timeline,1.4,51.46987,34.06639,48.01675,72.32646,23048
|
||||
frozen_head_initialized_lora_3_epochs,egospeak_full_timeline,1.6,50.91134,34.04663,46.85967,71.82772,23048
|
||||
frozen_head_initialized_lora_3_epochs,egospeak_full_timeline,1.8,50.54008,33.82003,46.15156,71.64864,23048
|
||||
frozen_head_initialized_lora_3_epochs,egospeak_full_timeline,2.0,50.30475,33.74069,45.80527,71.36829,23048
|
||||
frozen_head_initialized_lora_3_epochs,causal_valid_only,0.2,71.08773,53.71399,75.95995,83.58924,23045
|
||||
frozen_head_initialized_lora_3_epochs,causal_valid_only,0.4,65.20927,46.04420,69.65188,79.93173,23042
|
||||
frozen_head_initialized_lora_3_epochs,causal_valid_only,0.6,60.11088,40.44134,62.71649,77.17481,23039
|
||||
frozen_head_initialized_lora_3_epochs,causal_valid_only,0.8,56.36069,36.98112,56.62225,75.47870,23036
|
||||
frozen_head_initialized_lora_3_epochs,causal_valid_only,1.0,53.49230,35.10974,51.65037,73.71680,23033
|
||||
frozen_head_initialized_lora_3_epochs,causal_valid_only,1.2,52.51364,34.83634,49.83161,72.87296,23030
|
||||
frozen_head_initialized_lora_3_epochs,causal_valid_only,1.4,51.48201,34.06639,48.03402,72.34561,23027
|
||||
frozen_head_initialized_lora_3_epochs,causal_valid_only,1.6,50.92429,34.04874,46.87600,71.84812,23024
|
||||
frozen_head_initialized_lora_3_epochs,causal_valid_only,1.8,50.55369,33.82210,46.16732,71.67164,23021
|
||||
frozen_head_initialized_lora_3_epochs,causal_valid_only,2.0,50.31903,33.74274,45.82075,71.39361,23018
|
||||
|
@@ -0,0 +1,43 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"recorded_date": "2026-08-24",
|
||||
"status": "single_run_diagnostic_snapshot",
|
||||
"dataset": "EasyCom",
|
||||
"prediction_step_seconds": 0.2,
|
||||
"video_fps": 2,
|
||||
"metrics_percent": {
|
||||
"frozen_dense_cache_control": {
|
||||
"full_timeline_horizon_0.2_map": 71.69695,
|
||||
"full_timeline_mean_map": 58.16091,
|
||||
"causal_valid_mean_map": 58.17146
|
||||
},
|
||||
"frozen_states_sparse_supervision_50_epochs": {
|
||||
"full_timeline_mean_map": 57.20379,
|
||||
"causal_valid_mean_map": 57.21367,
|
||||
"full_timeline_horizon_map": [71.69194, 65.98617, 61.14072, 57.39854, 54.57513, 53.70320, 52.74547, 52.04826, 51.55154, 51.19693],
|
||||
"causal_valid_horizon_map": [71.69383, 65.98941, 61.14698, 57.40704, 54.58522, 53.71497, 52.75864, 52.06226, 51.56613, 51.21218]
|
||||
},
|
||||
"sparse_lora_epoch1": {
|
||||
"full_timeline_horizon_0.2_map": 67.40167,
|
||||
"full_timeline_mean_map": 56.52480,
|
||||
"causal_valid_mean_map": 56.53450
|
||||
},
|
||||
"frozen_head_initialized_lora_3_epochs": {
|
||||
"full_timeline_mean_map": 56.19608,
|
||||
"causal_valid_mean_map": 56.20535,
|
||||
"full_timeline_horizon_map": [71.08589, 65.20613, 60.10484, 56.35250, 53.48271, 52.50271, 51.46987, 50.91134, 50.54008, 50.30475],
|
||||
"causal_valid_horizon_map": [71.08773, 65.20927, 60.11088, 56.36069, 53.49230, 52.51364, 51.48201, 50.92429, 50.55369, 50.31903]
|
||||
},
|
||||
"sparse_lora_weights_only_epoch2": {
|
||||
"full_timeline_horizon_0.2_map": 65.55214,
|
||||
"full_timeline_mean_map": 54.19850,
|
||||
"causal_valid_mean_map": 54.20711
|
||||
}
|
||||
},
|
||||
"provenance": {
|
||||
"source": "User-reported remote console tables",
|
||||
"raw_metrics_json_archived": false,
|
||||
"exact_remote_feature_root_archived": false,
|
||||
"multi_seed": false
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user