Files

EasyCom frozen Qwen2.5-Omni-3B last-4 learned mixture

Recorded on 2026-08-26 from the user-reported remote training and infer_turn_head.py output. This is a single-run diagnostic result, not yet a multi-seed paper number.

Experiment contract

  • Dataset: EasyCom.
  • Test split: confirmed as EgoSpeak official Sessions 3, 2, and 1. Training is recorded as the script's default official Sessions 4--12; verify this against the remote checkpoint JSON because the supplied excerpt starts at epoch 24.
  • Backbone: frozen Qwen2.5-Omni-3B Thinker.
  • Streaming mode: open-stream replay with an 8-second causal-memory budget.
  • Input sampling: 2 FPS video and raw audio.
  • Prediction rate: 5 Hz, with ten future horizons from 0.2 to 2.0 seconds.
  • Readout: the query states from the final four Thinker layers, ordered oldest to newest, followed by a learned global softmax layer mixture and the same 10 x 3 anticipation MLP.
  • Training: head and four scalar layer logits only; Thinker remains frozen.
  • Evaluator: EgoSpeak full-timeline per-frame AP/mAP.

The remote artifact is named easycom_frozen_3b_8s_last4_learned_mix_50ep.pth, but the supplied log and the user report both show that the actual run ended at epoch 25/25. The 50ep suffix is therefore a naming error and must not be used as provenance.

Summary

All values below are percentages. The comparison baseline is the previously recorded frozen 3B, 8-second, open-stream, single-final-layer run evaluated on the same official test sessions.

Readout 0.2s full mAP Full mean mAP Delta at 0.2s Delta mean
Single final-layer query 71.99768 58.51937 -- --
Last-4 learned mixture 73.45085 60.53423 +1.45317 +2.01486

The last-4 readout improves every evaluated horizon. The largest gain is +2.41084 at 1.0 seconds; the smallest is still a positive +1.45317 at 0.2 seconds.

Per-horizon full-timeline results

Horizon Last-4 mAP Single-layer mAP Delta Background AP Wearer AP Others AP
0.2s 73.45085 71.99768 +1.45317 56.69385 78.35111 85.30760
0.4s 68.58645 66.74289 +1.84356 49.77891 73.48058 82.49988
0.6s 64.44213 62.38972 +2.05241 44.51563 68.39317 80.41758
0.8s 61.34852 59.12229 +2.22623 40.98328 64.20310 78.85917
1.0s 59.12464 56.71380 +2.41084 38.99833 60.75541 77.62018
1.2s 57.53667 55.36300 +2.17367 37.76815 58.15110 76.69076
1.4s 56.43710 54.21297 +2.22413 37.22701 56.12642 75.95786
1.6s 55.42035 53.43221 +1.98814 36.72606 54.14045 75.39454
1.8s 54.76334 52.84703 +1.91631 36.45931 52.80226 75.02845
2.0s 54.23227 52.37207 +1.86020 36.45292 51.56488 74.67902
Mean 60.53423 58.51937 +2.01486 41.56035 61.79685 78.24550

Averaged over horizons, the AP gains relative to the single-layer baseline are +1.92941 for background, +2.88478 for camera-wearer speaking, and +1.23041 for other-people speaking. The wearer class contributes the largest improvement, which is encouraging for the short-horizon turn-taking objective.

Learned layer mixture

Layer order Weight
Oldest of selected four 0.246384
Second 0.248327
Third 0.241774
Final layer 0.263515

The weights remain close to uniform: normalized entropy is 0.999619, the L1 distance from [0.25, 0.25, 0.25, 0.25] is only 0.02703, and the largest weight exceeds the smallest by only 0.021741. Consequently, this experiment supports multi-layer query readout, but does not yet show that learning the mixture is better than a fixed last-4 mean.

Interpretation and next controls

  1. This is the strongest recorded frozen-Omni EasyCom result in the project so far: 73.45085 at 0.2 seconds and 60.53423 mean mAP.
  2. The gain occurs at all horizons and all three classes, rather than coming from a single AP outlier.
  3. Because the learned weights are nearly uniform, the first required ablation is a fixed uniform last-4 mean with the same feature files, head recipe, epoch count, and seed.
  4. Then compare each of the four individual query layers. This separates layer ensembling from access to a specifically better intermediate layer.
  5. Repeat the single-layer, fixed-mean, and learned-mixture variants with at least three identical seeds before using the gain as a paper result.
  6. The supplied output contains no strict causal-valid table. Archive the remote metrics.json or rerun the evaluator before the publication table.
  7. The single-layer comparison came from an earlier run; its random seed and head-training duration must be matched in the fixed-mean ablation.

Provenance

  • Remote feature root: /data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
  • Remote checkpoint: checkpoints/easycom_frozen_3b_8s_last4_learned_mix_50ep.pth
  • Remote result directory: results/easycom_frozen_3b_8s_last4_learned_mix_50ep
  • Final training loss: 0.822849.
  • Raw console excerpt: raw_console.txt.
  • Machine-readable values: summary.json and metrics.csv.
  • Raw remote metrics.json archived locally: no.
  • Exact training-session list verified from the remote checkpoint JSON: no.
  • Multi-seed: no.