78 lines
3.5 KiB
Markdown
78 lines
3.5 KiB
Markdown
|
# EasyCom: sparse Thinker-LoRA weights-only continuation (epoch2 snapshot)
|
||
|
|
|
||
|
|
Recorded on **2026-08-24** from the remote EasyCom evaluation console. This is
|
||
|
|
a single-run experiment snapshot, not a multi-seed paper result.
|
||
|
|
|
||
|
|
## Experiment contract
|
||
|
|
|
||
|
|
- Dataset: EasyCom with EgoSpeak's official split.
|
||
|
|
- Train sessions: 7, 11, 10, 6, 9, 4, 8, 5, and 12.
|
||
|
|
- Test sessions: 3, 2, and 1 (`n=23048` full-timeline rows).
|
||
|
|
- Backbone: Qwen2.5-Omni-3B Thinker with LoRA plus the `10 x 3` turn head.
|
||
|
|
- Context: 8 seconds; 2 FPS video; predictions and labels at 5 Hz.
|
||
|
|
- Training sampling: transition-sparse anchors (ordinary 1 Hz plus dense 5 Hz
|
||
|
|
anchors near target transitions).
|
||
|
|
- Evaluation: dense 5 Hz AV cache on every test timestamp.
|
||
|
|
- Continuation: adapter and head weights loaded from the epoch1 checkpoint,
|
||
|
|
but optimizer moments and the learning-rate scheduler were reinitialized.
|
||
|
|
Therefore, "epoch2" here means a weights-only second training run, not an
|
||
|
|
exact optimizer-state resume.
|
||
|
|
- Reported checkpoint directory:
|
||
|
|
`checkpoints/omni_turn_lora_easycom_sparse_8s_epoch2_ddp4/`.
|
||
|
|
|
||
|
|
## Summary
|
||
|
|
|
||
|
|
All values are percentages.
|
||
|
|
|
||
|
|
| Checkpoint | Full-timeline mean mAP | Strict causal mean mAP |
|
||
|
|
|---|---:|---:|
|
||
|
|
| Frozen Thinker dense-cache control | 58.16091 | 58.17146 |
|
||
|
|
| Sparse LoRA epoch1 | 56.52480 | 56.53450 |
|
||
|
|
| Sparse LoRA weights-only epoch2 | **54.19850** | **54.20711** |
|
||
|
|
| Epoch2 minus epoch1 | -2.32630 | -2.32739 |
|
||
|
|
| Epoch2 minus frozen control | -3.96241 | -3.96435 |
|
||
|
|
|
||
|
|
The second run is materially worse than both epoch1 and the frozen control.
|
||
|
|
This is not consistent with normal evaluator noise.
|
||
|
|
|
||
|
|
## Per-horizon results
|
||
|
|
|
||
|
|
| Horizon | Full mAP | Strict mAP | Background AP | Wearer AP | Others AP |
|
||
|
|
|---:|---:|---:|---:|---:|---:|
|
||
|
|
| 0.2 s | 65.55214 | 65.55385 | 49.03788 | 65.76312 | 81.85542 |
|
||
|
|
| 0.4 s | 60.96517 | 60.96812 | 43.06922 | 61.14251 | 78.68378 |
|
||
|
|
| 0.6 s | 56.79766 | 56.80301 | 39.37427 | 55.10528 | 75.91344 |
|
||
|
|
| 0.8 s | 54.11741 | 54.12457 | 36.94546 | 51.56720 | 73.83955 |
|
||
|
|
| 1.0 s | 52.09538 | 52.10400 | 35.40706 | 48.38835 | 72.49075 |
|
||
|
|
| 1.2 s | 51.35516 | 51.36504 | 35.30874 | 46.86261 | 71.89412 |
|
||
|
|
| 1.4 s | 50.99774 | 51.00914 | 35.29891 | 45.48818 | 72.20613 |
|
||
|
|
| 1.6 s | 50.47830 | 50.49067 | 34.98109 | 44.74277 | 71.71105 |
|
||
|
|
| 1.8 s | 50.00999 | 50.02296 | 34.64341 | 43.86161 | 71.52496 |
|
||
|
|
| 2.0 s | 49.61605 | 49.62970 | 34.26290 | 43.14791 | 71.43735 |
|
||
|
|
|
||
|
|
The AP columns above are from the full-timeline protocol. Complete AP values
|
||
|
|
for both protocols are stored in [`metrics.csv`](metrics.csv), and
|
||
|
|
[`summary.json`](summary.json) stores the machine-readable aggregate result.
|
||
|
|
|
||
|
|
## Interpretation and decision
|
||
|
|
|
||
|
|
The largest class-specific regression is camera-wearer speaking. Relative to
|
||
|
|
epoch1, wearer AP falls from 68.77118 to 65.76312 at 0.2 seconds and from
|
||
|
|
47.53875 to 43.14791 at 2.0 seconds. Background and other-speaker AP decline
|
||
|
|
less at the long horizon. The pattern indicates negative transfer in the
|
||
|
|
wearer-speaking representation rather than a uniform metric offset.
|
||
|
|
|
||
|
|
Likely contributors are sparse-train/dense-evaluation distribution mismatch,
|
||
|
|
too much LoRA capacity for EasyCom, and restarting the learning-rate schedule
|
||
|
|
during weights-only continuation. Do not run a third continuation epoch with
|
||
|
|
the same recipe. Keep the frozen 8-second result as the main representation
|
||
|
|
baseline and next test the parameter-matched frozen overlap-working-memory
|
||
|
|
ablation.
|
||
|
|
|
||
|
|
## Provenance
|
||
|
|
|
||
|
|
- Source: user-reported `eval_omni_turn_lora.py` console output.
|
||
|
|
- Raw remote `metrics.json` archived here: no; values were transcribed.
|
||
|
|
- Multi-seed: no.
|
||
|
|
- Publication status: diagnostic negative result only.
|