wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add two-GPU AV producer for Omni LoRA

Browse Source
This commit is contained in:
Xin Wang
2026-08-23 15:29:26 +08:00
parent dc932cc165
commit c2162e1170
2 changed files with 273 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+103 -2
View File
@@ -40,8 +40,8 @@ classification head.
The full Thinker input path, embedded audio, direct video decoding,
FlashAttention-2, 5 Hz streaming extraction, cache replay, target alignment,
head training, and EgoSpeak-compatible evaluation have now been exercised on
EasyCom. Ego4D feature extraction and multi-seed EasyCom evaluation remain in
progress.
EasyCom. The first Ego4D 8-second versus 20-second context ablation has also
been recorded. Multi-seed evaluation remains in progress.
## Current EasyCom results
@@ -64,6 +64,69 @@ The full per-horizon and per-class snapshot, machine-readable CSV, provenance,
and follow-up checks are in
[`results/easycom_open_stream_vs_causal_prefix_2026-08-19/`](results/easycom_open_stream_vs_causal_prefix_2026-08-19/README.md).
### EasyCom 8-second versus 20-second context
An additional single-run open-stream comparison recorded on 2026-08-23 gives
**58.51937** full-timeline mean mAP with an 8-second context budget versus
**58.316** with 20 seconds, a difference of **+0.20337** points. The aggregate
difference is small and hides a clear horizon tradeoff: 8 seconds is stronger
at 0.2--0.8 seconds, whereas 20 seconds is stronger from 1.0 through 2.0
seconds. These runs must be repeated with identical additional seeds before
choosing a context budget for the paper.
The raw 8-second console output, machine-readable comparison, and detailed
interpretation are archived in
[`results/easycom_open_stream_8s_vs_20s_2026-08-23/`](results/easycom_open_stream_8s_vs_20s_2026-08-23/README.md).
## Current Ego4D context-length results
The following single-seed snapshot was recorded on 2026-08-22. Both variants
use open-stream replay, 2 FPS Decord video, 5 Hz predictions, the same frozen
Qwen2.5-Omni-3B Thinker and independently trained `10 x 3` heads. They differ
only in the rolling causal-memory budget: 8 seconds versus 20 seconds. Both
runs use the same EgoSpeak Ego4D train/test manifest and test timeline.
| Context | EgoSpeak full-timeline mean mAP | Strict causal mean mAP |
|---|---:|---:|
| 8 seconds | **65.5943** | **65.6920** |
| 20 seconds | 63.1408 | 63.2298 |
| 20s minus 8s | -2.4535 | -2.4622 |
The strict-causal per-horizon comparison is:
| Horizon | 8s mAP | 20s mAP | 20s minus 8s |
|---:|---:|---:|---:|
| 0.2s | **74.2213** | 71.7403 | -2.4810 |
| 0.4s | **71.0180** | 68.5945 | -2.4234 |
| 0.6s | **68.3746** | 65.9287 | -2.4459 |
| 0.8s | **66.3190** | 63.8606 | -2.4584 |
| 1.0s | **64.7470** | 62.3329 | -2.4140 |
| 1.2s | **63.6416** | 61.2290 | -2.4126 |
| 1.4s | **62.8697** | 60.4035 | -2.4662 |
| 1.6s | **62.3142** | 59.7870 | -2.5272 |
| 1.8s | **61.8705** | 59.3678 | -2.5027 |
| 2.0s | **61.5444** | 59.0541 | -2.4903 |
| Mean | **65.6920** | 63.2298 | -2.4622 |
For the exact full-timeline protocol, averaging class AP over the ten
horizons gives:
| Class | 8s mean AP | 20s mean AP | 20s minus 8s |
|---|---:|---:|---:|
| Background | **69.5063** | 67.2409 | -2.2653 |
| Camera wearer speaking | **59.4005** | 56.5784 | -2.8222 |
| Other people speaking | **67.8760** | 65.6030 | -2.2730 |
The 8-second representation is better at every horizon and for every class.
The approximately 2.4--2.5 point gap is nearly constant across anticipation
horizons, while the largest class degradation is for camera-wearer speech.
This pattern does not support a long-horizon benefit from the additional 12
seconds of context. A plausible hypothesis is that excessive history dilutes
locally predictive turn-taking cues in the final frozen Thinker state, but the
single-seed comparison cannot establish that mechanism. Both context settings
must be repeated with the same additional seeds before treating the difference
as a final paper result.
## Why Omni-only instead of Omni + RNN
EgoSpeak separately extracts visual and audio features and then uses GRU,
@@ -840,6 +903,44 @@ Recommended experimental procedure:
Class-balanced/focal loss, a learned turn-query token, and LoRA are later
ablations. They should not replace the frozen-Omni linear/MLP-head baseline.
## Planned Thinker-LoRA sampling and cache policy
This is a planned second-stage experiment, not a reported benchmark result.
Dense 5 Hz partial-prefix caching is fast during training but does not fit the
combined EasyCom and Ego4D storage budget. The 24-second Session 7 pilot used
0.115 GiB, which extrapolates to about 17.25 GiB per media hour. Caching only
complete 2-second history blocks and recomputing every current partial online
reduced storage, but the single-L20 pilot slowed 200 LoRA updates from 83 s
(2.39 update/s) to 254.085 s (0.787 update/s). Online AV preparation used
98.654 s for 163 partials and peak allocation was 9.56 GiB, so the bottleneck
is repeated preprocessing/modality encoding rather than memory capacity.
The proposed training sampler is therefore:
1. retain a uniform low-rate set of ordinary anchors, initially 1 Hz;
2. retain every 5 Hz anchor within a two-second band around annotated speaker
onset, offset, handover, and overlap-state transitions;
3. optionally add high-loss or high-uncertainty anchors after a first training
pass, while selecting them only from the training split;
4. cache exact partial AV embeddings only for these selected anchors, plus one
reusable complete embedding for every required 2-second history block;
5. enforce a 240 GiB cache target and a hard 300 GB upper bound, leaving room
for manifests, checkpoints, and extracted evaluation states.
Transition-centered selection is training-example sampling, not inference
look-ahead: model inputs must still end at the current time. Final extraction,
head training, and Table 2 evaluation remain on the complete 5 Hz timeline.
The paper must report the sampling rule and compare uniform-only sampling with
transition-enhanced sampling so that any gain is not attributed solely to
additional transition examples.
For a speed-only alternative, `train_omni_turn_lora.py` supports a two-GPU
producer-consumer benchmark. The producer GPU runs the frozen AV towers and
places current-tail embeddings in a bounded CPU queue while the consumer GPU
runs Thinker LoRA. This preserves exact online partial semantics; it changes
only scheduling. Compare its reported update rate and consumer queue-wait
time with the synchronous 0.787 update/s pilot before adopting it.
## EgoSpeak-compatible evaluation
Run inference on the official test sessions (`Session_3`, `Session_2`, and