icassp2027-experiment/results/easycom_dense_last8_qvo_frozen_head_epoch3_2026-08-25

EasyCom dense last-8 Thinker LoRA with a frozen head

Recorded on 2026-08-25 from user-reported remote evaluation output. This is a single-run diagnostic result, not a multi-seed paper number.

Experiment contract

• Model: Qwen2.5-Omni-3B Thinker.
• Inputs: dense cached AV phases, 2 FPS video, raw-audio tower embeddings, and an 8-second causal memory budget.
• Prediction/label rate: 5 Hz with ten horizons from 0.2 to 2.0 seconds.
• LoRA scope: q_proj, v_proj, and o_proj in Thinker layers 28--35 only.
• LoRA configuration: rank 8, alpha 16, dropout 0.05, learning rate 5e-6, weight decay 0.01.
• Head: the trained frozen 8-second head, loaded from checkpoints/easycom_open_stream_2fps_8s_head_seed0.pth and kept frozen.
• LoRA training: dense Sessions 4--11, global batch 16, five continuous epochs.
• Development selection: Session 12, primary metric 0.2-second full-timeline mAP and mean mAP as tie-breaker.
• Formal evaluation: epoch 3 was first evaluated on official EasyCom test Sessions 3, 2, and 1 after development selection. Epoch 1 was subsequently evaluated as a diagnostic before fixing the one-epoch full-training recipe.
• Selected checkpoint: epoch 3, step 11910.

The selected remote checkpoint was:

checkpoints/omni_turn_lora_dense_8s_last8_qvo_frozen_head_epoch5/
  checkpoint-step-11910/

The diagnostic epoch-1 checkpoint was:

checkpoints/omni_turn_lora_dense_8s_last8_qvo_frozen_head_epoch5/
  checkpoint-step-3970/

Development selection on Session 12

All values are percentages.

Epoch	Step	0.2s mAP	Mean mAP
1	3970	70.80918	63.06487
2	7940	70.96876	63.09318
3	11910	70.99153	63.01294
4	15880	70.87364	62.85465
5	19850	70.83634	62.81744

Epoch 3 was selected by the preregistered 0.2-second criterion. The five-run range is only 0.18235 points at 0.2 seconds and 0.27574 mean points, indicating that the restricted low-learning-rate LoRA is much more stable than the prior full-Thinker sparse-LoRA runs. Epochs 4--5 nevertheless show mild overfitting.

Official Sessions 1--3 results

Epoch 1 and epoch 3 are close at 0.2 seconds, but epoch 1 preserves the longer horizons better and therefore has the higher mean mAP.

Checkpoint	0.2s mAP	Mean mAP	0.2s delta vs frozen	Mean delta vs frozen
Frozen dense control	71.69695	58.16091	--	--
LoRA epoch 1 / step 3970	71.59585	58.01500	-0.10110	-0.14591
LoRA epoch 3 / step 11910	71.65000	57.88833	-0.04695	-0.27258

Epoch 1 is only 0.05415 below epoch 3 at 0.2 seconds and is 0.12667 higher in mean mAP. Relative to epoch 3, it improves every horizon from 0.6 through 2.0 seconds. This supports fixing one epoch and retraining once on all official training Sessions 4--12, without development checkpoint selection.

Epoch 1 / step 3970

Horizon	LoRA mAP	Frozen dense mAP	Delta	Background AP	Wearer AP	Others AP
0.2s	71.59585	71.69695	-0.10110	54.40604	76.39165	83.98987
0.4s	66.37987	66.46468	-0.08481	47.56313	70.58803	80.98845
0.6s	61.91962	62.03503	-0.11541	42.34205	64.79174	78.62507
0.8s	58.62537	58.77409	-0.14872	39.04030	59.81340	77.02241
1.0s	56.30013	56.43606	-0.13593	37.19759	55.95535	75.74744
1.2s	54.86526	55.00306	-0.13780	36.31219	53.41217	74.87142
1.4s	53.59386	53.73114	-0.13728	35.39635	51.13887	74.24635
1.6s	52.86682	53.04042	-0.17360	35.19829	49.67869	73.72348
1.8s	52.23019	52.42454	-0.19435	34.71486	48.59750	73.37822
2.0s	51.77300	52.00315	-0.23015	34.68133	47.63528	73.00237
Mean	58.01500	58.16091	-0.14591	39.68521	57.80027	76.55951

Epoch 3 / step 11910

The fair control is the frozen Thinker evaluated from the same dense AV cache path with the same frozen head. Delta is selected LoRA minus that control.

Horizon	LoRA mAP	Frozen dense mAP	Delta	Background AP	Wearer AP	Others AP
0.2s	71.65000	71.69695	-0.04695	54.59245	76.13410	84.22345
0.4s	66.38669	66.46468	-0.07799	47.57127	70.36283	81.22596
0.6s	61.87698	62.03503	-0.15805	42.33591	64.49864	78.79640
0.8s	58.51356	58.77409	-0.26053	38.97583	59.42986	77.13498
1.0s	56.13447	56.43606	-0.30159	37.07515	55.55303	75.77523
1.2s	54.69993	55.00306	-0.30313	36.20847	53.01781	74.87352
1.4s	53.41760	53.73114	-0.31354	35.30111	50.71685	74.23484
1.6s	52.64243	53.04042	-0.39799	35.00690	49.21422	73.70616
1.8s	52.00887	52.42454	-0.41567	34.55009	48.10873	73.36779
2.0s	51.55275	52.00315	-0.45040	34.55502	47.13447	72.96877
Mean	57.88833	58.16091	-0.27258	39.61722	57.41705	76.63071

The class-mean comparison is:

Class	LoRA mean AP	Frozen dense mean AP	Delta
Background	39.61722	39.68434	-0.06712
Camera wearer speaking	57.41705	58.04423	-0.62718
Other people speaking	76.63071	76.75417	-0.12346

At 0.2 seconds specifically, background and other-person AP increase by +0.10051 and +0.09021, but wearer AP falls by -0.33157; the net mAP is therefore -0.04695. The negative gap grows almost monotonically with horizon and reaches -0.45040 at 2.0 seconds.

Interpretation

1. Restricted LoRA nearly eliminates the short-horizon collapse but does not create positive transfer. The prior frozen-head initialized sparse LoRA scored 71.08589 at 0.2 seconds and 56.19608 mean; the new recipe improves those by +0.56411 and +1.69225, respectively, but remains below the frozen dense control.
2. The remaining negative transfer is horizon-dependent. The 0.2-second representation is essentially preserved, while longer anticipation is progressively damaged. This is inconsistent with a simple global calibration error.
3. Camera-wearer speech is the main failure mode. Its average AP decreases by 0.62718 points, much more than background or other-person speech.
4. More epochs are not justified. Epoch 1 is already within 0.05415 points of epoch 3 at 0.2 seconds and is 0.12667 points better in mean mAP on the official test sessions. The remaining official-split experiment therefore fixes one epoch and trains on Sessions 4--12 without checkpoint selection.
5. This checkpoint is an ablation, not a headline improvement. It does not beat frozen Omni, the 58.51937 open-stream frozen-8s result, or EgoSpeak GRU A+V at 60.6 mean mAP.

The next defensible post-training diagnostic is adapter scaling/interpolation on a clean development protocol. It should not be tuned from the already-seen Sessions 1--3 result. For publication, retrain the frozen head without the chosen development session; the reused frozen head was originally trained on the official training split and may already have seen Session 12.

Provenance and limitations

• Source: user-reported remote eval_omni_turn_lora.py console tables.
• Raw remote metrics.json archived here: no.
• Strict causal-valid table supplied: no.
• Exact remote git revision confirmed: no; the relevant implementation commits include 7acab44, 7fdda29, and aa4278e.
• Multi-seed: no.
• Test-use caveat: epoch 1 was inspected after the epoch-3 test result; their comparison is diagnostic rather than untouched test-set model selection.
• Development caveat: the frozen head may have been trained with Session 12; this is adequate for diagnosis but is not a clean publication dev split.
• Publication status: negative-transfer ablation only.