Archive EasyCom sparse supervision results

2026-08-24 12:02:49 +08:00
parent edd37dc36a
commit b7856ece4d
8 changed files with 430 additions and 0 deletions
@@ -0,0 +1,94 @@
+# EasyCom: sparse supervision versus Thinker LoRA
+
+Recorded on **2026-08-24** from user-reported remote evaluation tables. These
+are single-run diagnostic results, not final multi-seed paper numbers.
+
+## Experiment contract
+
+- Dataset and split: EasyCom with EgoSpeak's official nine training sessions
+  and test Sessions 3, 2, and 1.
+- Prediction rate: 5 Hz; ten future horizons from 0.2 to 2.0 seconds.
+- Video sampling: 2 FPS.
+- Evaluator: EgoSpeak full-timeline AP/mAP plus strict causal-valid AP/mAP.
+- Sparse policy: ordinary anchors at 1 Hz plus 5 Hz anchors around annotated
+  target transitions.
+- Evaluation is dense at every 5 Hz test timestamp.
+- Frozen sparse run: previously extracted frozen Thinker states; only the head
+  is trained for 50 epochs on sparse target indices.
+- LoRA three-epoch run: initialized with the trained frozen head, then updated
+  the Thinker LoRA and head for three epochs.
+
+The remote `metrics.json`, exact checkpoint configuration, git revision, and
+feature-root path were not copied into this repository. The context/feature
+identity with the frozen 8-second control must therefore be verified before
+publication.
+
+## Summary
+
+All values are percentages.
+
+| Experiment | 0.2s full mAP | Full mean mAP | Strict mean mAP |
+|---|---:|---:|---:|
+| Frozen Thinker dense-cache control | 71.69695 | 58.16091 | 58.17146 |
+| Frozen states + sparse supervision, 50 epochs | **71.69194** | **57.20379** | **57.21367** |
+| Sparse LoRA epoch 1 | 67.40167 | 56.52480 | 56.53450 |
+| Frozen-head initialized LoRA, 3 epochs | **71.08589** | **56.19608** | **56.20535** |
+| Sparse LoRA weights-only epoch 2 | 65.55214 | 54.19850 | 54.20711 |
+
+The epoch-2 continuation is archived separately in
+[`../easycom_omni_lora_sparse_epoch2_2026-08-24/`](../easycom_omni_lora_sparse_epoch2_2026-08-24/README.md).
+
+## Frozen states with sparse supervision
+
+| Horizon | Full mAP | Strict mAP | Full background AP | Full wearer AP | Full others AP |
+|---:|---:|---:|---:|---:|---:|
+| 0.2s | 71.69194 | 71.69383 | 54.64681 | 76.09602 | 84.33299 |
+| 0.4s | 65.98617 | 65.98941 | 46.86427 | 70.48309 | 80.61116 |
+| 0.6s | 61.14072 | 61.14698 | 41.30997 | 64.40092 | 77.71127 |
+| 0.8s | 57.39854 | 57.40704 | 37.50173 | 59.07507 | 75.61882 |
+| 1.0s | 54.57513 | 54.58522 | 35.47209 | 54.41110 | 73.84220 |
+| 1.2s | 53.70320 | 53.71497 | 35.16647 | 52.41903 | 73.52409 |
+| 1.4s | 52.74547 | 52.75864 | 34.64200 | 50.53203 | 73.06238 |
+| 1.6s | 52.04826 | 52.06226 | 34.36442 | 49.18784 | 72.59253 |
+| 1.8s | 51.55154 | 51.56613 | 34.13135 | 48.17972 | 72.34356 |
+| 2.0s | 51.19693 | 51.21218 | 33.99937 | 47.47712 | 72.11430 |
+| Mean | **57.20379** | **57.21367** | — | — | — |
+
+## Frozen-head initialized LoRA after three epochs
+
+| Horizon | Full mAP | Strict mAP | Full background AP | Full wearer AP | Full others AP |
+|---:|---:|---:|---:|---:|---:|
+| 0.2s | 71.08589 | 71.08773 | 53.71399 | 75.95995 | 83.58373 |
+| 0.4s | 65.20613 | 65.20927 | 46.04420 | 69.65188 | 79.92232 |
+| 0.6s | 60.10484 | 60.11088 | 40.44134 | 62.71065 | 77.16253 |
+| 0.8s | 56.35250 | 56.36069 | 36.98112 | 56.61255 | 75.46383 |
+| 1.0s | 53.48271 | 53.49230 | 35.10974 | 51.63824 | 73.70016 |
+| 1.2s | 52.50271 | 52.51364 | 34.83634 | 49.81663 | 72.85516 |
+| 1.4s | 51.46987 | 51.48201 | 34.06639 | 48.01675 | 72.32646 |
+| 1.6s | 50.91134 | 50.92429 | 34.04663 | 46.85967 | 71.82772 |
+| 1.8s | 50.54008 | 50.55369 | 33.82003 | 46.15156 | 71.64864 |
+| 2.0s | 50.30475 | 50.31903 | 33.74069 | 45.80527 | 71.36829 |
+| Mean | **56.19608** | **56.20535** | — | — | — |
+
+## 0.2-second diagnosis
+
+Sparse supervision alone changes 0.2-second full-timeline mAP by only
+`-0.00501` relative to the frozen dense-cache control. The three-epoch LoRA
+run changes it by `-0.61106`. The much larger epoch-1 and weights-only epoch-2
+regressions show that short-horizon degradation is primarily associated with
+updating the Thinker representation, not with dropping ordinary supervision
+anchors.
+
+The next required control is a 2-by-2 checkpoint evaluation: frozen Thinker
+with frozen head, frozen Thinker with LoRA-trained head, LoRA Thinker with
+frozen head, and LoRA Thinker with LoRA-trained head. Select LoRA checkpoints
+on a training-session development split using 0.2-second mAP rather than the
+official test sessions.
+
+## Provenance
+
+- Source: user-reported remote `infer_turn_head.py` and
+  `eval_omni_turn_lora.py` console tables.
+- Raw remote metrics archived: no.
+- Multi-seed: no.
+- Publication status: diagnostic only.