wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add competing-risk evaluation and align ICASSP story

Browse Source
This commit is contained in:
Xin Wang
2026-09-10 11:12:02 +08:00
parent 0abefd6f77
commit 4b7d2c0c20
30 changed files with 3420 additions and 724 deletions
Download Patch File Download Diff File Expand all files Collapse all files
results/easycom_roma_frozen_last4_2026-08-27/README.md
+123
View File
@@ -0,0 +1,123 @@
# EasyCom frozen-ROMA last-four-layer readout study
Recorded on **2026-08-27** from three user-reported remote evaluations. These
are single-seed diagnostic results on official EasyCom test Sessions 1--3, not
test-independent paper numbers.
## Shared experiment contract
- Dataset: EasyCom; EgoSpeak full-timeline evaluator on Sessions 1--3.
- Backbone checkpoint: ROMA, based on Qwen2.5-Omni-7B.
- Backbone use: frozen representation extraction; no end-to-end ROMA update.
- Streaming path: this project's 8-second Qwen-style open-stream replay, 2 FPS
video, one prediction every 200 ms.
- Saved state: final four query states, shape `[T, 4, 3584]`.
- Prediction target: ten future horizons from 0.2 to 2.0 seconds, three
EgoSpeak AP classes.
ROMA's released model is trained with one-second aligned streaming units, but
these features were extracted with this project's matched Qwen open-stream
protocol. The result therefore tests the **frozen ROMA representation under
our protocol**; it does not reproduce ROMA's own streaming training recipe.
## Main result
| Readout | Epochs | Approx. fusion parameters | 0.2s mAP | Mean mAP |
|---|---:|---:|---:|---:|
| Last-4 concat-MLP | 1 | 51.4M | 68.27145 | 58.89219 |
| Last-4 concat-MLP | 4 | 51.4M | 68.65486 | 56.52425 |
| Last-4 fixed mean | 4 | 0 | **69.21863** | **58.90402** |
The fixed uniform mean is the strongest ROMA readout. It is effectively tied
with the one-epoch concat model in mean mAP (`+0.01183`) while gaining
`+0.94718` at 0.2 seconds. The 51.4M-parameter fusion is unnecessary here.
## Per-horizon mAP
| Horizon | Concat 1ep | Concat 4ep | Fixed mean 4ep |
|---:|---:|---:|---:|
| 0.2s | 68.27145 | 68.65486 | **69.21863** |
| 0.4s | 65.29126 | 64.32941 | **65.67454** |
| 0.6s | 62.15965 | 60.41845 | **62.52327** |
| 0.8s | **59.95476** | 57.35198 | 59.92499 |
| 1.0s | **58.25407** | 55.17851 | 58.14859 |
| 1.2s | **56.83849** | 53.70716 | 56.61984 |
| 1.4s | 55.65764 | 52.59370 | **55.68495** |
| 1.6s | **54.91836** | 51.83772 | 54.52641 |
| 1.8s | **54.20725** | 50.70519 | 53.67463 |
| 2.0s | **53.36895** | 50.46558 | 53.04431 |
| **Mean** | 58.89219 | 56.52425 | **58.90402** |
## Mean class AP over ten horizons
| Readout | Background | Camera wearer | Other people |
|---|---:|---:|---:|
| Concat 1ep | 42.17735 | **57.38537** | 77.11384 |
| Concat 4ep | 40.19574 | 52.75190 | 76.62513 |
| Fixed mean 4ep | **42.19246** | 57.26979 | **77.24980** |
Extending concat training from one to four epochs raises 0.2-second mAP by
only `+0.38341` but reduces mean mAP by `-2.36794`. Camera-wearer mean AP drops
by `-4.63347`, much more than background (`-1.98161`) or other people
(`-0.48871`). This is the same overfitting pattern observed with the 3B
concat-MLP, amplified by ROMA's wider 3584-dimensional state.
Replacing concat-MLP with a fixed mean at four epochs restores `+4.51789`
camera-wearer AP and `+2.37977` mean mAP. This isolates readout capacity as the
proximate cause of the four-epoch collapse.
## Comparison with the strongest frozen Qwen2.5-Omni-3B controls
| Frozen representation/readout | 0.2s mAP | Mean mAP |
|---|---:|---:|
| Qwen-3B final-layer query, 8s | 71.99768 | 58.51937 |
| Qwen-3B last-4 scalar mixture | **73.45085** | 60.53423 |
| Qwen-3B last-4 concat-MLP, 4ep | 73.42020 | **60.87144** |
| ROMA last-4 fixed mean, 4ep | 69.21863 | 58.90402 |
| ROMA last-4 concat-MLP, 4ep | 68.65486 | 56.52425 |
The same concat design and epoch count gives a strong diagnostic: ROMA is
`-4.76534` at 0.2 seconds and `-4.34719` in mean mAP relative to Qwen-3B.
It is not a parameter-matched attribution test, because the 3584-dimensional
ROMA fusion has 51.4M parameters versus about 16.8M for Qwen-3B. Together with
the fixed-mean result, it shows that ROMA's proactive streaming adaptation is
not automatically useful for 5 Hz egocentric future-speaker anticipation, but
does not yet isolate checkpoint adaptation from width and protocol.
The fixed-mean ROMA result is not directly architecture-matched to the 3B
scalar-mixture/concat models. An exact Qwen-3B fixed-mean four-epoch control is
still required. A second clean attribution test is ROMA versus its original
Qwen2.5-Omni-7B base using the same last-four stack and fixed-mean head.
## Interpretation and decision
1. **Do not continue tuning the ROMA concat head.** Its capacity is much too
high for nine EasyCom training conversations.
2. **Do not conclude that ROMA itself is universally worse.** The experiment
changes checkpoint family and uses a protocol different from ROMA's native
one-second training units.
3. **The practical EasyCom backbone remains frozen Qwen2.5-Omni-3B.** Its
last-four readouts are stronger, cheaper, and already validated throughout
the current pipeline.
4. **The next method should change readout structure and supervision.** The
strongest hypothesis is a low-capacity semantic query state fused with
recent 200 ms payload tokens, followed by a structured future-activity
projection rather than ten independent horizon classifiers.
## Validity caveats
- Only user-reported full-timeline metrics were available; strict
causal-valid tables and remote `metrics.json` files were not archived.
- Results are single-seed and Sessions 1--3 were inspected repeatedly.
- Exact optimizer settings and the remote checkpoint filenames for all three
runs were not supplied here.
- The train split is assumed to follow the project's Sessions 4--12 contract;
verify each remote command before using the values in a paper table.
- ROMA's official one-second streaming-unit path was not reproduced.
Machine-readable values are in [`summary.json`](summary.json), and a formatted
workbook with per-horizon metrics and comparison deltas is in
[`metrics.xlsx`](metrics.xlsx).
The literature-backed diagnosis and proposed ICASSP research plan are in
[`reports/roma_negative_transfer_and_future_floor_projection_2026-08-27.md`](../../reports/roma_negative_transfer_and_future_floor_projection_2026-08-27.md).