Files

EasyCom: sparse Thinker-LoRA weights-only continuation (epoch2 snapshot)

Recorded on 2026-08-24 from the remote EasyCom evaluation console. This is a single-run experiment snapshot, not a multi-seed paper result.

Experiment contract

  • Dataset: EasyCom with EgoSpeak's official split.
  • Train sessions: 7, 11, 10, 6, 9, 4, 8, 5, and 12.
  • Test sessions: 3, 2, and 1 (n=23048 full-timeline rows).
  • Backbone: Qwen2.5-Omni-3B Thinker with LoRA plus the 10 x 3 turn head.
  • Context: 8 seconds; 2 FPS video; predictions and labels at 5 Hz.
  • Training sampling: transition-sparse anchors (ordinary 1 Hz plus dense 5 Hz anchors near target transitions).
  • Evaluation: dense 5 Hz AV cache on every test timestamp.
  • Continuation: adapter and head weights loaded from the epoch1 checkpoint, but optimizer moments and the learning-rate scheduler were reinitialized. Therefore, "epoch2" here means a weights-only second training run, not an exact optimizer-state resume.
  • Reported checkpoint directory: checkpoints/omni_turn_lora_easycom_sparse_8s_epoch2_ddp4/.

Summary

All values are percentages.

Checkpoint Full-timeline mean mAP Strict causal mean mAP
Frozen Thinker dense-cache control 58.16091 58.17146
Sparse LoRA epoch1 56.52480 56.53450
Sparse LoRA weights-only epoch2 54.19850 54.20711
Epoch2 minus epoch1 -2.32630 -2.32739
Epoch2 minus frozen control -3.96241 -3.96435

The second run is materially worse than both epoch1 and the frozen control. This is not consistent with normal evaluator noise.

Per-horizon results

Horizon Full mAP Strict mAP Background AP Wearer AP Others AP
0.2 s 65.55214 65.55385 49.03788 65.76312 81.85542
0.4 s 60.96517 60.96812 43.06922 61.14251 78.68378
0.6 s 56.79766 56.80301 39.37427 55.10528 75.91344
0.8 s 54.11741 54.12457 36.94546 51.56720 73.83955
1.0 s 52.09538 52.10400 35.40706 48.38835 72.49075
1.2 s 51.35516 51.36504 35.30874 46.86261 71.89412
1.4 s 50.99774 51.00914 35.29891 45.48818 72.20613
1.6 s 50.47830 50.49067 34.98109 44.74277 71.71105
1.8 s 50.00999 50.02296 34.64341 43.86161 71.52496
2.0 s 49.61605 49.62970 34.26290 43.14791 71.43735

The AP columns above are from the full-timeline protocol. Complete AP values for both protocols are stored in metrics.csv, and summary.json stores the machine-readable aggregate result.

Interpretation and decision

The largest class-specific regression is camera-wearer speaking. Relative to epoch1, wearer AP falls from 68.77118 to 65.76312 at 0.2 seconds and from 47.53875 to 43.14791 at 2.0 seconds. Background and other-speaker AP decline less at the long horizon. The pattern indicates negative transfer in the wearer-speaking representation rather than a uniform metric offset.

Likely contributors are sparse-train/dense-evaluation distribution mismatch, too much LoRA capacity for EasyCom, and restarting the learning-rate schedule during weights-only continuation. Do not run a third continuation epoch with the same recipe. Keep the frozen 8-second result as the main representation baseline and next test the parameter-matched frozen overlap-working-memory ablation.

Provenance

  • Source: user-reported eval_omni_turn_lora.py console output.
  • Raw remote metrics.json archived here: no; values were transcribed.
  • Multi-seed: no.
  • Publication status: diagnostic negative result only.