Archive EasyCom sparse supervision results

2026-08-24 12:02:49 +08:00
parent edd37dc36a
commit b7856ece4d
8 changed files with 430 additions and 0 deletions
@@ -0,0 +1,94 @@
+# EasyCom: sparse supervision versus Thinker LoRA
+
+Recorded on **2026-08-24** from user-reported remote evaluation tables. These
+are single-run diagnostic results, not final multi-seed paper numbers.
+
+## Experiment contract
+
+- Dataset and split: EasyCom with EgoSpeak's official nine training sessions
+  and test Sessions 3, 2, and 1.
+- Prediction rate: 5 Hz; ten future horizons from 0.2 to 2.0 seconds.
+- Video sampling: 2 FPS.
+- Evaluator: EgoSpeak full-timeline AP/mAP plus strict causal-valid AP/mAP.
+- Sparse policy: ordinary anchors at 1 Hz plus 5 Hz anchors around annotated
+  target transitions.
+- Evaluation is dense at every 5 Hz test timestamp.
+- Frozen sparse run: previously extracted frozen Thinker states; only the head
+  is trained for 50 epochs on sparse target indices.
+- LoRA three-epoch run: initialized with the trained frozen head, then updated
+  the Thinker LoRA and head for three epochs.
+
+The remote `metrics.json`, exact checkpoint configuration, git revision, and
+feature-root path were not copied into this repository. The context/feature
+identity with the frozen 8-second control must therefore be verified before
+publication.
+
+## Summary
+
+All values are percentages.
+
+| Experiment | 0.2s full mAP | Full mean mAP | Strict mean mAP |
+|---|---:|---:|---:|
+| Frozen Thinker dense-cache control | 71.69695 | 58.16091 | 58.17146 |
+| Frozen states + sparse supervision, 50 epochs | **71.69194** | **57.20379** | **57.21367** |
+| Sparse LoRA epoch 1 | 67.40167 | 56.52480 | 56.53450 |
+| Frozen-head initialized LoRA, 3 epochs | **71.08589** | **56.19608** | **56.20535** |
+| Sparse LoRA weights-only epoch 2 | 65.55214 | 54.19850 | 54.20711 |
+
+The epoch-2 continuation is archived separately in
+[`../easycom_omni_lora_sparse_epoch2_2026-08-24/`](../easycom_omni_lora_sparse_epoch2_2026-08-24/README.md).
+
+## Frozen states with sparse supervision
+
+| Horizon | Full mAP | Strict mAP | Full background AP | Full wearer AP | Full others AP |
+|---:|---:|---:|---:|---:|---:|
+| 0.2s | 71.69194 | 71.69383 | 54.64681 | 76.09602 | 84.33299 |
+| 0.4s | 65.98617 | 65.98941 | 46.86427 | 70.48309 | 80.61116 |
+| 0.6s | 61.14072 | 61.14698 | 41.30997 | 64.40092 | 77.71127 |
+| 0.8s | 57.39854 | 57.40704 | 37.50173 | 59.07507 | 75.61882 |
+| 1.0s | 54.57513 | 54.58522 | 35.47209 | 54.41110 | 73.84220 |
+| 1.2s | 53.70320 | 53.71497 | 35.16647 | 52.41903 | 73.52409 |
+| 1.4s | 52.74547 | 52.75864 | 34.64200 | 50.53203 | 73.06238 |
+| 1.6s | 52.04826 | 52.06226 | 34.36442 | 49.18784 | 72.59253 |
+| 1.8s | 51.55154 | 51.56613 | 34.13135 | 48.17972 | 72.34356 |
+| 2.0s | 51.19693 | 51.21218 | 33.99937 | 47.47712 | 72.11430 |
+| Mean | **57.20379** | **57.21367** | — | — | — |
+
+## Frozen-head initialized LoRA after three epochs
+
+| Horizon | Full mAP | Strict mAP | Full background AP | Full wearer AP | Full others AP |
+|---:|---:|---:|---:|---:|---:|
+| 0.2s | 71.08589 | 71.08773 | 53.71399 | 75.95995 | 83.58373 |
+| 0.4s | 65.20613 | 65.20927 | 46.04420 | 69.65188 | 79.92232 |
+| 0.6s | 60.10484 | 60.11088 | 40.44134 | 62.71065 | 77.16253 |
+| 0.8s | 56.35250 | 56.36069 | 36.98112 | 56.61255 | 75.46383 |
+| 1.0s | 53.48271 | 53.49230 | 35.10974 | 51.63824 | 73.70016 |
+| 1.2s | 52.50271 | 52.51364 | 34.83634 | 49.81663 | 72.85516 |
+| 1.4s | 51.46987 | 51.48201 | 34.06639 | 48.01675 | 72.32646 |
+| 1.6s | 50.91134 | 50.92429 | 34.04663 | 46.85967 | 71.82772 |
+| 1.8s | 50.54008 | 50.55369 | 33.82003 | 46.15156 | 71.64864 |
+| 2.0s | 50.30475 | 50.31903 | 33.74069 | 45.80527 | 71.36829 |
+| Mean | **56.19608** | **56.20535** | — | — | — |
+
+## 0.2-second diagnosis
+
+Sparse supervision alone changes 0.2-second full-timeline mAP by only
+`-0.00501` relative to the frozen dense-cache control. The three-epoch LoRA
+run changes it by `-0.61106`. The much larger epoch-1 and weights-only epoch-2
+regressions show that short-horizon degradation is primarily associated with
+updating the Thinker representation, not with dropping ordinary supervision
+anchors.
+
+The next required control is a 2-by-2 checkpoint evaluation: frozen Thinker
+with frozen head, frozen Thinker with LoRA-trained head, LoRA Thinker with
+frozen head, and LoRA Thinker with LoRA-trained head. Select LoRA checkpoints
+on a training-session development split using 0.2-second mAP rather than the
+official test sessions.
+
+## Provenance
+
+- Source: user-reported remote `infer_turn_head.py` and
+  `eval_omni_turn_lora.py` console tables.
+- Raw remote metrics archived: no.
+- Multi-seed: no.
+- Publication status: diagnostic only.
@@ -0,0 +1,41 @@
+experiment,protocol,horizon_seconds,map,background_ap,camera_wearer_speaking_ap,other_people_speaking_ap,num_steps
+frozen_states_sparse_supervision_50_epochs,egospeak_full_timeline,0.2,71.69194,54.64681,76.09602,84.33299,23048
+frozen_states_sparse_supervision_50_epochs,egospeak_full_timeline,0.4,65.98617,46.86427,70.48309,80.61116,23048
+frozen_states_sparse_supervision_50_epochs,egospeak_full_timeline,0.6,61.14072,41.30997,64.40092,77.71127,23048
+frozen_states_sparse_supervision_50_epochs,egospeak_full_timeline,0.8,57.39854,37.50173,59.07507,75.61882,23048
+frozen_states_sparse_supervision_50_epochs,egospeak_full_timeline,1.0,54.57513,35.47209,54.41110,73.84220,23048
+frozen_states_sparse_supervision_50_epochs,egospeak_full_timeline,1.2,53.70320,35.16647,52.41903,73.52409,23048
+frozen_states_sparse_supervision_50_epochs,egospeak_full_timeline,1.4,52.74547,34.64200,50.53203,73.06238,23048
+frozen_states_sparse_supervision_50_epochs,egospeak_full_timeline,1.6,52.04826,34.36442,49.18784,72.59253,23048
+frozen_states_sparse_supervision_50_epochs,egospeak_full_timeline,1.8,51.55154,34.13135,48.17972,72.34356,23048
+frozen_states_sparse_supervision_50_epochs,egospeak_full_timeline,2.0,51.19693,33.99937,47.47712,72.11430,23048
+frozen_states_sparse_supervision_50_epochs,causal_valid_only,0.2,71.69383,54.64681,76.09602,84.33868,23045
+frozen_states_sparse_supervision_50_epochs,causal_valid_only,0.4,65.98941,46.86427,70.48309,80.62088,23042
+frozen_states_sparse_supervision_50_epochs,causal_valid_only,0.6,61.14698,41.30997,64.40704,77.72392,23039
+frozen_states_sparse_supervision_50_epochs,causal_valid_only,0.8,57.40704,37.50173,59.08557,75.63383,23036
+frozen_states_sparse_supervision_50_epochs,causal_valid_only,1.0,54.58522,35.47209,54.42458,73.85900,23033
+frozen_states_sparse_supervision_50_epochs,causal_valid_only,1.2,53.71497,35.16647,52.43571,73.54273,23030
+frozen_states_sparse_supervision_50_epochs,causal_valid_only,1.4,52.75864,34.64200,50.55135,73.08256,23027
+frozen_states_sparse_supervision_50_epochs,causal_valid_only,1.6,52.06226,34.36659,49.20607,72.61411,23024
+frozen_states_sparse_supervision_50_epochs,causal_valid_only,1.8,51.56613,34.13348,48.19712,72.36779,23021
+frozen_states_sparse_supervision_50_epochs,causal_valid_only,2.0,51.21218,34.00148,47.49396,72.14111,23018
+frozen_head_initialized_lora_3_epochs,egospeak_full_timeline,0.2,71.08589,53.71399,75.95995,83.58373,23048
+frozen_head_initialized_lora_3_epochs,egospeak_full_timeline,0.4,65.20613,46.04420,69.65188,79.92232,23048
+frozen_head_initialized_lora_3_epochs,egospeak_full_timeline,0.6,60.10484,40.44134,62.71065,77.16253,23048
+frozen_head_initialized_lora_3_epochs,egospeak_full_timeline,0.8,56.35250,36.98112,56.61255,75.46383,23048
+frozen_head_initialized_lora_3_epochs,egospeak_full_timeline,1.0,53.48271,35.10974,51.63824,73.70016,23048
+frozen_head_initialized_lora_3_epochs,egospeak_full_timeline,1.2,52.50271,34.83634,49.81663,72.85516,23048
+frozen_head_initialized_lora_3_epochs,egospeak_full_timeline,1.4,51.46987,34.06639,48.01675,72.32646,23048
+frozen_head_initialized_lora_3_epochs,egospeak_full_timeline,1.6,50.91134,34.04663,46.85967,71.82772,23048
+frozen_head_initialized_lora_3_epochs,egospeak_full_timeline,1.8,50.54008,33.82003,46.15156,71.64864,23048
+frozen_head_initialized_lora_3_epochs,egospeak_full_timeline,2.0,50.30475,33.74069,45.80527,71.36829,23048
+frozen_head_initialized_lora_3_epochs,causal_valid_only,0.2,71.08773,53.71399,75.95995,83.58924,23045
+frozen_head_initialized_lora_3_epochs,causal_valid_only,0.4,65.20927,46.04420,69.65188,79.93173,23042
+frozen_head_initialized_lora_3_epochs,causal_valid_only,0.6,60.11088,40.44134,62.71649,77.17481,23039
+frozen_head_initialized_lora_3_epochs,causal_valid_only,0.8,56.36069,36.98112,56.62225,75.47870,23036
+frozen_head_initialized_lora_3_epochs,causal_valid_only,1.0,53.49230,35.10974,51.65037,73.71680,23033
+frozen_head_initialized_lora_3_epochs,causal_valid_only,1.2,52.51364,34.83634,49.83161,72.87296,23030
+frozen_head_initialized_lora_3_epochs,causal_valid_only,1.4,51.48201,34.06639,48.03402,72.34561,23027
+frozen_head_initialized_lora_3_epochs,causal_valid_only,1.6,50.92429,34.04874,46.87600,71.84812,23024
+frozen_head_initialized_lora_3_epochs,causal_valid_only,1.8,50.55369,33.82210,46.16732,71.67164,23021
+frozen_head_initialized_lora_3_epochs,causal_valid_only,2.0,50.31903,33.74274,45.82075,71.39361,23018
@@ -0,0 +1,43 @@
+{
+  "schema_version": 1,
+  "recorded_date": "2026-08-24",
+  "status": "single_run_diagnostic_snapshot",
+  "dataset": "EasyCom",
+  "prediction_step_seconds": 0.2,
+  "video_fps": 2,
+  "metrics_percent": {
+    "frozen_dense_cache_control": {
+      "full_timeline_horizon_0.2_map": 71.69695,
+      "full_timeline_mean_map": 58.16091,
+      "causal_valid_mean_map": 58.17146
+    },
+    "frozen_states_sparse_supervision_50_epochs": {
+      "full_timeline_mean_map": 57.20379,
+      "causal_valid_mean_map": 57.21367,
+      "full_timeline_horizon_map": [71.69194, 65.98617, 61.14072, 57.39854, 54.57513, 53.70320, 52.74547, 52.04826, 51.55154, 51.19693],
+      "causal_valid_horizon_map": [71.69383, 65.98941, 61.14698, 57.40704, 54.58522, 53.71497, 52.75864, 52.06226, 51.56613, 51.21218]
+    },
+    "sparse_lora_epoch1": {
+      "full_timeline_horizon_0.2_map": 67.40167,
+      "full_timeline_mean_map": 56.52480,
+      "causal_valid_mean_map": 56.53450
+    },
+    "frozen_head_initialized_lora_3_epochs": {
+      "full_timeline_mean_map": 56.19608,
+      "causal_valid_mean_map": 56.20535,
+      "full_timeline_horizon_map": [71.08589, 65.20613, 60.10484, 56.35250, 53.48271, 52.50271, 51.46987, 50.91134, 50.54008, 50.30475],
+      "causal_valid_horizon_map": [71.08773, 65.20927, 60.11088, 56.36069, 53.49230, 52.51364, 51.48201, 50.92429, 50.55369, 50.31903]
+    },
+    "sparse_lora_weights_only_epoch2": {
+      "full_timeline_horizon_0.2_map": 65.55214,
+      "full_timeline_mean_map": 54.19850,
+      "causal_valid_mean_map": 54.20711
+    }
+  },
+  "provenance": {
+    "source": "User-reported remote console tables",
+    "raw_metrics_json_archived": false,
+    "exact_remote_feature_root_archived": false,
+    "multi_seed": false
+  }
+}