wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update README with verification status of EasyCom components and add results from recent experiments comparing open-stream replay and causal-prefix methods. Introduce new metrics CSV and detailed README for the EasyCom results, including current performance metrics and required follow-up actions before publication.

Browse Source
This commit is contained in:
Xin Wang
2026-08-19 11:29:27 +08:00
parent 5fa61eed23
commit 92bf93c96b
4 changed files with 233 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+31 -9
View File
@@ -30,17 +30,39 @@ classification head.
| File | Status | Purpose |
|---|---|---|
| `smoke_test.py` | Verified on one NVIDIA L20 | Text and 12-second EasyCom audio-video inference |
| `easycom_manifest.py` | Implemented, awaiting dataset-side run | Discover one-minute clips, preserve session timecodes, and check 5 Hz targets |
| `easycom_manifest.py` | Verified on EasyCom | Discover one-minute clips, preserve session timecodes, and check 5 Hz targets |
| `qwen_omni_features.py` | Implemented, not executed here | Load only the 3B Thinker and extract fused states from one AV window |
| `streaming_turn_model.py` | Implemented | Produce one causal Omni state every 200 ms, with causal-prefix and open-stream modes |
| `TurnPredictionHead` | Defined, untrained | Map `[2048]` to `[10, 3]` logits |
| `train_turn_head.py` | Implemented | Train the frozen-Omni anticipation head with the official EasyCom split |
| `infer_turn_head.py` | Implemented | Save EgoSpeak-format predictions and reproduce per-horizon AP/mAP |
| `streaming_turn_model.py` | Verified on EasyCom | Produce one causal Omni state every 200 ms, with causal-prefix and open-stream modes |
| `TurnPredictionHead` | Trained and evaluated | Map `[2048]` to `[10, 3]` logits |
| `train_turn_head.py` | Verified on EasyCom | Train the frozen-Omni anticipation head with the official EasyCom split |
| `infer_turn_head.py` | Verified on EasyCom | Save EgoSpeak-format predictions and reproduce per-horizon AP/mAP |
The official full model, Thinker input path, embedded audio, video decoding,
and FlashAttention-2 have passed a 12-second EasyCom smoke test on one NVIDIA
L20. Streaming extraction, cache equivalence, target alignment, training, and
paper metrics remain unverified.
The full Thinker input path, embedded audio, direct video decoding,
FlashAttention-2, 5 Hz streaming extraction, cache replay, target alignment,
head training, and EgoSpeak-compatible evaluation have now been exercised on
EasyCom. Ego4D feature extraction and multi-seed EasyCom evaluation remain in
progress.
## Current EasyCom results
The current single-run comparison was recorded on 2026-08-19. Both modes use
the frozen Qwen2.5-Omni-3B Thinker, 2 FPS video, 5 Hz predictions, a 20-second
causal context budget, the same `10 x 3` head architecture, and EgoSpeak's
official EasyCom train/test split.
| Representation | EgoSpeak full-timeline mean mAP | Strict causal mean mAP |
|---|---:|---:|
| Open-stream replay | 58.316 | 58.326 |
| Causal-prefix | 58.830 | 58.841 |
Causal-prefix is 0.514 mAP points higher in the exact Table 2 protocol, but
the difference is concentrated at 0.2--0.6 seconds. Open-stream is slightly
higher at 1.6--2.0 seconds. These are not yet final paper values because the
head-training variance has not been measured across multiple seeds.
The full per-horizon and per-class snapshot, machine-readable CSV, provenance,
and follow-up checks are in
[`results/easycom_open_stream_vs_causal_prefix_2026-08-19/`](results/easycom_open_stream_vs_causal_prefix_2026-08-19/README.md).
## Why Omni-only instead of Omni + RNN