EasyCom frozen-ROMA last-four-layer readout study

Recorded on 2026-08-27 from three user-reported remote evaluations. These are single-seed diagnostic results on official EasyCom test Sessions 1--3, not test-independent paper numbers.

Shared experiment contract

Dataset: EasyCom; EgoSpeak full-timeline evaluator on Sessions 1--3.
Backbone checkpoint: ROMA, based on Qwen2.5-Omni-7B.
Backbone use: frozen representation extraction; no end-to-end ROMA update.
Streaming path: this project's 8-second Qwen-style open-stream replay, 2 FPS video, one prediction every 200 ms.
Saved state: final four query states, shape [T, 4, 3584].
Prediction target: ten future horizons from 0.2 to 2.0 seconds, three EgoSpeak AP classes.

ROMA's released model is trained with one-second aligned streaming units, but these features were extracted with this project's matched Qwen open-stream protocol. The result therefore tests the frozen ROMA representation under our protocol; it does not reproduce ROMA's own streaming training recipe.

Main result

Readout	Epochs	Approx. fusion parameters	0.2s mAP	Mean mAP
Last-4 concat-MLP	1	51.4M	68.27145	58.89219
Last-4 concat-MLP	4	51.4M	68.65486	56.52425
Last-4 fixed mean	4	0	69.21863	58.90402

The fixed uniform mean is the strongest ROMA readout. It is effectively tied with the one-epoch concat model in mean mAP (+0.01183) while gaining +0.94718 at 0.2 seconds. The 51.4M-parameter fusion is unnecessary here.

Per-horizon mAP

Horizon	Concat 1ep	Concat 4ep	Fixed mean 4ep
0.2s	68.27145	68.65486	69.21863
0.4s	65.29126	64.32941	65.67454
0.6s	62.15965	60.41845	62.52327
0.8s	59.95476	57.35198	59.92499
1.0s	58.25407	55.17851	58.14859
1.2s	56.83849	53.70716	56.61984
1.4s	55.65764	52.59370	55.68495
1.6s	54.91836	51.83772	54.52641
1.8s	54.20725	50.70519	53.67463
2.0s	53.36895	50.46558	53.04431
Mean	58.89219	56.52425	58.90402

Mean class AP over ten horizons

Readout	Background	Camera wearer	Other people
Concat 1ep	42.17735	57.38537	77.11384
Concat 4ep	40.19574	52.75190	76.62513
Fixed mean 4ep	42.19246	57.26979	77.24980

Extending concat training from one to four epochs raises 0.2-second mAP by only +0.38341 but reduces mean mAP by -2.36794. Camera-wearer mean AP drops by -4.63347, much more than background (-1.98161) or other people (-0.48871). This is the same overfitting pattern observed with the 3B concat-MLP, amplified by ROMA's wider 3584-dimensional state.

Replacing concat-MLP with a fixed mean at four epochs restores +4.51789 camera-wearer AP and +2.37977 mean mAP. This isolates readout capacity as the proximate cause of the four-epoch collapse.

Comparison with the strongest frozen Qwen2.5-Omni-3B controls

Frozen representation/readout	0.2s mAP	Mean mAP
Qwen-3B final-layer query, 8s	71.99768	58.51937
Qwen-3B last-4 scalar mixture	73.45085	60.53423
Qwen-3B last-4 concat-MLP, 4ep	73.42020	60.87144
ROMA last-4 fixed mean, 4ep	69.21863	58.90402
ROMA last-4 concat-MLP, 4ep	68.65486	56.52425

The same concat design and epoch count gives a strong diagnostic: ROMA is -4.76534 at 0.2 seconds and -4.34719 in mean mAP relative to Qwen-3B. It is not a parameter-matched attribution test, because the 3584-dimensional ROMA fusion has 51.4M parameters versus about 16.8M for Qwen-3B. Together with the fixed-mean result, it shows that ROMA's proactive streaming adaptation is not automatically useful for 5 Hz egocentric future-speaker anticipation, but does not yet isolate checkpoint adaptation from width and protocol.

The fixed-mean ROMA result is not directly architecture-matched to the 3B scalar-mixture/concat models. An exact Qwen-3B fixed-mean four-epoch control is still required. A second clean attribution test is ROMA versus its original Qwen2.5-Omni-7B base using the same last-four stack and fixed-mean head.

Interpretation and decision

Do not continue tuning the ROMA concat head. Its capacity is much too high for nine EasyCom training conversations.
Do not conclude that ROMA itself is universally worse. The experiment changes checkpoint family and uses a protocol different from ROMA's native one-second training units.
The practical EasyCom backbone remains frozen Qwen2.5-Omni-3B. Its last-four readouts are stronger, cheaper, and already validated throughout the current pipeline.
The next method should change readout structure and supervision. The strongest hypothesis is a low-capacity semantic query state fused with recent 200 ms payload tokens, followed by a structured future-activity projection rather than ten independent horizon classifiers.

Validity caveats

Only user-reported full-timeline metrics were available; strict causal-valid tables and remote metrics.json files were not archived.
Results are single-seed and Sessions 1--3 were inspected repeatedly.
Exact optimizer settings and the remote checkpoint filenames for all three runs were not supplied here.
The train split is assumed to follow the project's Sessions 4--12 contract; verify each remote command before using the values in a paper table.
ROMA's official one-second streaming-unit path was not reproduced.

Machine-readable values are in summary.json, and a formatted workbook with per-horizon metrics and comparison deltas is in metrics.xlsx.

The literature-backed diagnosis and proposed ICASSP research plan are in reports/roma_negative_transfer_and_future_floor_projection_2026-08-27.md.

5.9 KiB Raw Blame History