wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'main' of https://gitea.xiaowang.eu.org/wx44wx/icassp2027-experiment

Browse Source
This commit is contained in:
Xin Wang
2026-08-17 22:46:44 +08:00
parent 5ee371f0ae 8145be4108
commit a6dea1529a
15 changed files with 3379 additions and 489 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+272 -103
View File
@@ -32,9 +32,10 @@ classification head.
| `smoke_test.py` | Verified on one NVIDIA L20 | Text and 12-second EasyCom audio-video inference |
| `easycom_manifest.py` | Implemented, awaiting dataset-side run | Discover one-minute clips, preserve session timecodes, and check 5 Hz targets |
| `qwen_omni_features.py` | Implemented, not executed here | Load only the 3B Thinker and extract fused states from one AV window |
| `streaming_turn_model.py` | Prototype implemented, not executed here | Produce one causal Omni state every 200 ms, with reference and KV-cache modes |
| `streaming_turn_model.py` | Implemented | Produce one causal Omni state every 200 ms, with causal-prefix and open-stream modes |
| `TurnPredictionHead` | Defined, untrained | Map `[2048]` to `[10, 3]` logits |
| Head training and EgoSpeak evaluation script | Not yet implemented | Align saved states with `target_perframe`, train the head, and reproduce Table 2 mAP |
| `train_turn_head.py` | Implemented | Train the frozen-Omni anticipation head with the official EasyCom split |
| `infer_turn_head.py` | Implemented | Save EgoSpeak-format predictions and reproduce per-horizon AP/mAP |
The official full model, Thinker input path, embedded audio, video decoding,
and FlashAttention-2 have passed a 12-second EasyCom smoke test on one NVIDIA
@@ -284,13 +285,17 @@ Session_1.json clip order, exact T, crop and sampling provenance
Session_1.ffconcat reproducible dense concatenation order
```
The script fails rather than silently padding or truncating when
`sum(source_frames) / 4 != target.shape[0]`.
The target length must equal the number of complete 5 Hz intervals,
`floor(sum(source_frames) / 4)`. If a session ends with one to three residual
20 Hz frames, the target-derived duration discards only that incomplete final
interval; the discarded frame count is recorded in the sidecar.
Each clip's decoded video and audio are trimmed to the exact duration implied
by its video frame count before concatenation. This prevents AAC priming/tail
padding from accumulating at the 24 clip boundaries and causing progressive
audio/video drift.
audio/video drift. The session-global visual grid is padded defensively at EOF
and then trimmed by frame index, so it contains exactly
`ceil(target_steps / 5 * video_fps)` frames on every FFmpeg version.
On the remote L20 machine, use its own dataset paths. For example:
@@ -312,33 +317,112 @@ script and reuse the existing MP4 without encoding it again:
python prepare_easycom_session.py ... --fps 2 --reuse-existing
```
## What Qwen's native 2-second chunk means
## What Qwen's native 2-second streaming design means
Qwen2.5-Omni is **not restricted to receiving audio once every two seconds**.
It can receive an arbitrary-duration AV input. The two-second value describes
internal processing:
Qwen2.5-Omni was designed for **block-wise multimodal prefill**. Its
[technical report](https://arxiv.org/html/2503.20215) states that the audio
and visual encoders are modified for block-wise processing and that audio
self-attention is performed in independent two-second blocks. The released
[3B configuration](https://huggingface.co/Qwen/Qwen2.5-Omni-3B/raw/main/config.json)
uses `seconds_per_chunk=2`, `position_id_per_seconds=25`, and audio
`n_window=100`.
1. the audio encoder divides acoustic features into approximately two-second
attention blocks;
2. audio and video tokens are interleaved in two-second temporal groups before
entering the Thinker.
The audio encoder attention inside a block is non-causal. Therefore this is
invalid for online evaluation:
Conceptually, a native online input is organized as follows:
```text
encode the complete [0, 2.0 s] block
-> read a purported state at 0.2 s
open one user/media turn
encode/append AV [0,2] -> commit Thinker KV
encode/append AV [2,4] -> commit Thinker KV
encode/append AV [4,6] -> commit Thinker KV
...
close the media turn -> generate text and/or speech
```
That early state can contain audio from 0.2--2.0 seconds. At prediction time
`t`, the model must receive only audio and video with timestamps `<= t`.
Within each temporal group, video tokens precede the aligned audio tokens.
TMRoPE assigns 25 temporal positions per second, or 50 positions per native
two-second group, so audio and video share a physical-time coordinate system.
Completed blocks are appended to one causal Thinker sequence; the long-range
history is modeled by the Thinker rather than by cross-block full attention in
the modality encoders.
This architecture is block-causal, not frame-causal. Attention inside an
audio block is non-causal. It is therefore invalid to encode the complete
`[0,2]` block and claim that its representation at 0.2 seconds saw no future:
```text
encode complete [0,2] AV -> read state at 0.2 s # future leakage
```
For this project's 5 Hz targets, the current incomplete block must instead be
encoded only through the prediction time. These 0.2-second partial-block
readouts are an extension made for EgoSpeak; the Qwen paper does not define a
canonical hidden state every 200 ms.
The public Transformers and local Qwen web-demo APIs normally receive a
completed media object and call the processor/model once. They retain the
model's block-wise masks and token organization, but do not expose the full
persistent live-session implementation used by the hosted real-time product.
## Native Qwen streaming versus `causal-prefix`
The two extraction modes are both causal with respect to raw observations,
but they define different model inputs. `causal-prefix` is a sliding-window
reference invented for this experiment; it is not Qwen's native streaming
execution model.
| Property | `causal-prefix` | Native Qwen streaming idea |
|---|---|---|
| Session | New inference at every 0.2 s | One open media turn |
| Media organization | One complete `[max(0,t-M),t]` item | Fixed source-aligned 2 s blocks |
| Modality encoding | Re-encode the complete window | Encode/commit each completed block once |
| Thinker KV | Start empty for every state | Append to persistent KV |
| TMRoPE | Recomputed from each window origin | Advances with the stream |
| History | Exact rolling `M` seconds | Accumulates until a serving/context policy intervenes |
| 5 Hz readout | Close/read a new turn each time | Not specified by the original model |
The difference becomes especially important after a sliding window fills. At
`t=20.8` with `M=20`, `causal-prefix` constructs one new media item covering:
```text
[0.8,20.8]
-> encoder blocks [0.8,2.8], [2.8,4.8], ..., [18.8,20.8]
```
At `t=21.0`, every encoder boundary moves again:
```text
[1.0,21.0]
-> encoder blocks [1.0,3.0], [3.0,5.0], ..., [19.0,21.0]
```
Native streaming keeps the source-aligned partition instead:
```text
[0,2], [2,4], ..., [18,20], current partial [20,20.8]
```
Because Qwen's audio encoder uses block-local non-causal attention, moving a
block boundary changes the representation even when most raw samples are the
same. Window-relative TMRoPE, convolution/padding boundaries, chat closure,
and BF16/attention execution add further differences. This is why exact KV
replay can be numerically identical to its own from-scratch replay while still
differing from `causal-prefix`; the modes are evaluating different token and
encoder histories.
Use the modes according to the scientific question:
- use `open-stream-replay` for the primary online experiment, training, and
deployment-aligned evaluation;
- use `causal-prefix` as an expensive continuous-window reference and
short-clip ablation;
- never train a head on one representation and deploy it on the other without
retraining, because their feature distributions are not interchangeable.
## Causal 5 Hz extraction modes
`streaming_turn_model.py` implements three modes.
`streaming_turn_model.py` implements two modes.
### 1. `causal-prefix`: correctness and semantic baseline
### 1. `causal-prefix`: continuous-window reference
At every 200 ms, independently process the causal context ending at the
current time:
@@ -353,7 +437,8 @@ Properties:
- one continuous media window per prediction;
- no cross-forward cache;
- very expensive because overlapping context is recomputed;
- preferred reference when checking semantics and causality.
- useful for checking continuous sliding-window semantics and causality;
- not the native Qwen block-streaming execution model.
Example:
@@ -368,57 +453,7 @@ python streaming_turn_model.py \
--output features/easycom_20s_prefix.pt
```
### 2. `commit-replay`: closed-turn compatibility baseline
The extractor retains Thinker KV for completed two-second media blocks. During
the unfinished block it branches from the same committed cache and evaluates
only the available prefix:
```text
committed historical KV + current block [0.0, 0.2] -> q_t
committed historical KV + current block [0.0, 0.4] -> q_t
...
committed historical KV + current block [0.0, 2.0] -> q_t
-> commit media KV
```
The temporary readout/assistant query is never committed. Only the completed
media block enters long-term cache. This avoids recomputing the full history,
but still re-encodes at most the current two-second prefix.
Example with numerical checking:
```bash
python streaming_turn_model.py \
--video /path/to/easycom_session.mp4 \
--use-audio-in-video \
--mode commit-replay \
--start 0 \
--end 20 \
--memory-seconds 20 \
--verify-every 5 \
--verify-continuous-every 5 \
--output features/easycom_20s_stream.pt
```
`--verify-every 5` compares cached processing with a from-scratch replay of the
same sequence of two-second media turns. The saved cosine similarity should be
extremely close to 1. If it is not, the TMRoPE/cache path is invalid and must
not be used for a paper result.
`--verify-continuous-every 5` is a different check: it compares the cached
sequence of two-second media turns with a single continuous causal window.
This comparison need not be numerically identical, but it measures whether the
input organization materially changes the representation. Save and report the
two check types separately (`cache_replay` and `continuous_window_semantics`).
There is an important experimental distinction: `commit-replay` represents
history as a sequence of two-second media turns. Its numerical check proves
cache equivalence to that same turn sequence, not equivalence to one continuous
20-second video placeholder. Compare it against `causal-prefix` on identical
clips and report the representation if it changes mAP.
### 3. `open-stream-replay`: recommended efficient mode
### 2. `open-stream-replay`: recommended efficient mode
This mode keeps one user/media turn open across the whole bounded-memory
segment. The first block contributes the chat and AV opening markers once;
@@ -435,6 +470,18 @@ open AV prefix
AV/chat closure -> never committed; readout branch only
```
This matches the core Qwen design choices: fixed two-second AV groups, one
open media turn, completed-block commits, and incremental Thinker KV. It makes
two practical changes for this dataset:
1. the current 0--2 second partial block is evaluated speculatively every
0.2 seconds and discarded unless it reaches a complete block boundary;
2. the KV history is bounded and rebuilt from recent complete blocks rather
than accumulating for an entire multi-minute session.
It should therefore be described as **bounded Qwen-style open streaming**, not
as bit-for-bit reproduction of an unbounded official serving session.
The audio encoder accepts a shorter final chunk with padding and an
actual-length mask. For the two-frame visual patch requirement, strict-grid
mode duplicates only the most recent past frame when necessary.
@@ -465,29 +512,26 @@ required even though artificial user-turn boundaries have been removed.
## Whole-session prediction and the 20-second limit
Twenty seconds is a practical memory/context setting for the 3090, not the
maximum video duration. The complete session is processed sequentially and a
maximum video duration and not a limit specified by Qwen. The original
append-only idea eventually reaches both linear KV-memory growth and the
Thinker's 32,768-token context limit, so an unbounded multi-minute stream is
not practical. The complete session is processed sequentially and a
prediction is emitted every 0.2 seconds.
The legacy `commit-replay` prototype resets Thinker KV at each non-overlapping
20-second boundary:
```text
session: 0 -------- 20 -------- 40 -------- 60 ...
cache: [segment 1] [segment 2] [segment 3] ...
output: continuous predictions across the entire session
```
This preserves complete output coverage but limits available history after a
reset. Report the memory length and additionally evaluate the first two seconds
after each reset versus all other timesteps. Rolling KV eviction is a possible
future optimization, but requires correct multimodal position rebasing.
`open-stream-replay` instead uses blockwise rolling memory. With a 20-second
`open-stream-replay` uses blockwise rolling memory. With a 20-second
budget it rebuilds the open cache from the latest nine completed blocks
(18 seconds) when a new block starts, then adds the current 0--2 second tail.
Its effective history therefore varies from 18 to 20 seconds and never drops
to an empty 0.2-second context at a 20-second boundary.
For a short native-stream fidelity check, set `--memory-seconds` to at least
the tested clip duration. For example, the options
`--start 0 --end 40 --memory-seconds 40` prevent a rolling rebuild during that
40-second clip.
Do not extend this policy to complete EasyCom/Ego4D sessions: KV memory grows
linearly, and this implementation clones the cache for speculative partial
blocks and readout branches.
For the paper experiment, drive the output length from the preparation sidecar
rather than a floating MP4 duration:
@@ -517,6 +561,10 @@ the following block. It duplicates the most recent available frame when the
visual tower needs an even frame count. Audio is loaded once at 16 kHz and
sliced at the same causal boundary. The processor still receives
`use_audio_in_video=True`, so Qwen's audio/video token interleaving is retained.
Long runs use `tqdm` and update the progress bar after every completed state,
showing percentage, media time, elapsed time, ETA, and throughput. Pass
`--no-progress` to disable it. The Thinker-only path also suppresses Qwen's
irrelevant speech-output system prompt warning.
Before a full session, run only the first four seconds and inspect both check
types:
@@ -536,17 +584,85 @@ python streaming_turn_model.py \
--output features/Session_1_first_4s_check.pt
```
<<<<<<< HEAD
Do not launch all sessions unless `cache_replay` cosine is nearly 1. For
`open-stream-replay`, a materially lower `continuous_window_semantics` cosine
means that the custom payload stitching is not equivalent to one continuous
Qwen input; it must not be dismissed as a media-turn-boundary effect because
this mode intentionally keeps a single media turn open.
=======
Do not launch all sessions unless `cache_replay` cosine is nearly 1. The
`continuous_window_semantics` cosine answers a different question and may be
lower because blockwise media preprocessing changes the representation.
>>>>>>> 8145be410866dd0675038d92347aea34d24b84d5
Video can be sampled at 2 FPS while predictions remain at 5 Hz. Audio and the
causal boundary update every 200 ms; consecutive prediction times may use the
same most-recent visual frame. A frame with timestamp later than `t` must never
be included.
## Direct extraction without a prepared 2 FPS MP4
`streaming_turn_model_directly.py` removes the offline video-transcoding
requirement. It reads original frame PTS with Decord, constructs a global
2 FPS lattice, maps every lattice point to the latest source frame whose PTS
is not later than that point, and passes explicit frame lists to Qwen. Audio is
decoded once to 16-kHz mono and sliced at exact causal boundaries. Therefore
no FFmpeg `fps` filter or second H.264/AAC encode can alter the experiment.
Validate one Ego4D clip without loading Qwen:
```bash
python streaming_turn_model_directly.py \
--video /data/wangx/Datasets/ego4d/v1/clips/CLIP_UID.mp4 \
--target /data/wangx/Datasets/ego4d/ego4d_target_frame/CLIP_UID.npy \
--video-fps 2 \
--media-only
```
Run a short native persistent-KV check:
```bash
python streaming_turn_model_directly.py \
--model-id /data/wangx/Models/Qwen2.5-Omni-3B \
--video /data/wangx/Datasets/ego4d/v1/clips/CLIP_UID.mp4 \
--end 40 \
--mode open-stream-replay \
--memory-seconds 0 \
--video-fps 2 \
--flash-attention-2 \
--device cuda:0 \
--output features/direct/CLIP_UID_first_40s.pt
```
Here `--memory-seconds 0` means that completed-block Thinker KV persists for
the entire requested interval. Do not use this setting blindly for complete
multi-minute recordings: it has linear KV growth and remains subject to the
Thinker's context limit. A full bounded-memory dataset run should explicitly
use the experimental budget, for example `--memory-seconds 20`.
EasyCom can be read directly as the dense concatenation of retained clips;
filename timecode gaps are ignored and the rightmost 1920x1080 region is kept:
```bash
python streaming_turn_model_directly.py \
--model-id /path/to/Qwen2.5-Omni-3B \
--easycom-session-dir /path/to/Main/Video_Compressed/Session_1 \
--target target_frame/Session_1.npy \
--mode open-stream-replay \
--memory-seconds 20 \
--video-fps 2 \
--flash-attention-2 \
--device cuda:0 \
--output features/direct/Session_1.pt
```
Matching `states.shape[0]` to the target is necessary but not sufficient.
The direct reader also validates EasyCom's 20-Hz source inventory, source PTS
monotonicity, dense duration, crop bounds, audio sample count, and causal
frame-to-grid mapping. The original prepare scripts remain useful only as an
optional decode-speed/reproducibility cache.
## Saved streaming tensors
Each `.pt` file contains:
@@ -556,7 +672,7 @@ Each `.pt` file contains:
- `label_indices`: `[0, ..., N-1]`, direct `target_perframe` row alignment;
- `valid_10_horizon_steps`: `N-10`, because the last ten current rows do not
have all future labels;
- `mode`: `causal-prefix`, `commit-replay`, or `open-stream-replay`;
- `mode`: `causal-prefix` or `open-stream-replay`;
- `stream_representation`: explicit input-organization metadata;
- `config`: step, native block, memory, FPS, and pixel settings;
- `verification`: cached-versus-exact numerical checks;
@@ -580,7 +696,8 @@ LayerNorm(2048)
-> reshape [10, 3]
```
`streaming_turn_model.py` exposes:
`easycom_head.py` defines the head and loss; they remain re-exported by
`streaming_turn_model.py` for compatibility:
```python
from streaming_turn_model import (
@@ -589,16 +706,37 @@ from streaming_turn_model import (
)
```
Recommended initial procedure:
The supplied LSTR config uses Adam, 50 epochs, learning rate `7e-5`, weight
decay `5e-5`, dropout `0.2`, and 10 warmup epochs followed by cosine decay.
`train_turn_head.py` uses those defaults. Its batch size is 512 causal states,
not LSTR's 16 temporal windows, because the two model inputs have different
shapes.
Train once on the official nine sessions with a fixed 50-epoch protocol:
```bash
python train_turn_head.py \
--feature-root features/open_2fps \
--target-root target_frame \
--output checkpoints/open_2fps_head.pth \
--device cuda:0
```
The default loss is `lstr-mce`, exactly matching LSTR's MCE treatment of a
multi-positive `[0,1,1]` row: it sums both active negative-log-softmax terms
without normalizing the target. The previous L2-normalized objective belongs
to MiniROAD and is available only as the explicit
`--loss miniroad-normalized` ablation.
Recommended experimental procedure:
1. Freeze Qwen and extract all training-session states offline.
2. Verify state/label time offset and truncate only the final ten invalid
current positions.
3. Keep EgoSpeak targets as float multi-positive tensors `[N, 10, 3]`; never
convert them with `argmax`.
4. Train with `egospeak_multihorizon_loss`. It exactly mirrors MiniROAD's
anticipation objective: L2-normalize each three-class target, then compute
its weighted negative log-softmax.
4. For comparison with the supplied LSTR shell, train with the default
`egospeak_multihorizon_loss`/`lstr-mce` objective.
5. Select hyperparameters using only training sessions. For EasyCom, reserve a
training session for development or use session-level cross-validation;
never tune on Sessions 1--3.
@@ -610,7 +748,22 @@ ablations. They should not replace the frozen-Omni linear/MLP-head baseline.
## EgoSpeak-compatible evaluation
For every valid current timestep, produce probabilities shaped:
Run inference on the official test sessions (`Session_3`, `Session_2`, and
`Session_1`):
```bash
python infer_turn_head.py \
--feature-root features/open_2fps \
--target-root target_frame \
--checkpoint checkpoints/open_2fps_head.pth \
--output-dir results/open_2fps_head \
--device cuda:0
```
This writes `pred/Session_X.npy` with the exact LSTR/EgoSpeak layout
`[target_time, class, future_horizon]`, i.e. `[T,3,10]`, plus `metrics.json`.
For every valid current timestep, the head itself produces probabilities
shaped:
```text
[number of timesteps, 10 future horizons, 3 classes]
@@ -626,8 +779,22 @@ Then reproduce EgoSpeak's Table 2 procedure:
5. average the ten horizon mAP values for the final score.
With EgoSpeak's `ignore_index=-100`, the released evaluator does not remove
class index zero, so background is included. Reuse the same implementation or
verify a reimplementation numerically.
class index zero, so background is included. The script reports two protocols:
- `egospeak_full_timeline`: the same full `[T,3,10]` array and evaluator as
LSTR, with zeros where a prediction would require pre-session context;
- `causal_valid_only`: excludes only those unavailable first `h` rows for
horizon `h`. This is the primary strictly causal score.
The difference involves at most the first ten rows of each session, but both
are saved so the comparison is explicit.
The convenience wrapper mirrors EgoSpeak's train/inference shells:
```bash
chmod +x run_easycom_head_experiment.sh
CUDA_VISIBLE_DEVICES=0 ./run_easycom_head_experiment.sh all
```
Reference Table 2 targets from the EgoSpeak paper:
@@ -706,7 +873,7 @@ pip install -r requirements.txt
```
The implementation is pinned to `transformers==4.52.3` because the
commit/replay path uses version-specific Thinker internals. On Linux,
open-stream replay path uses version-specific Thinker internals. On Linux,
`qwen-omni-utils[decord]` is recommended for faster video decoding. On a
compatible NVIDIA GPU, optionally install `flash-attn` and pass
`--flash-attention-2`.
@@ -737,8 +904,9 @@ Do not launch full EasyCom/Ego4D extraction until all of these pass:
time `t`;
- `states` and `target_perframe` lengths/time origins are reconciled and the
chosen label offset is saved;
- commit/replay cosine equivalence is near 1 on short clips;
- commit/replay and causal-prefix predictions are compared on identical clips;
- open-stream cache replay is exactly equivalent to its from-scratch replay on
short clips;
- open-stream and causal-prefix predictions are compared on identical clips;
- a 20-second run fits the 3090 with the selected pixel budget and attention
implementation;
- saved outputs contain exactly ten future horizons in the order 0.2--2.0 s;
@@ -751,7 +919,8 @@ Once these checks pass, the first publishable experiment is:
```text
frozen Qwen2.5-Omni-3B Thinker
+ trained 10x3 head
+ 20-second causal memory
+ bounded Qwen-style 2-second open streaming
+ 20-second rolling causal memory
+ full-session 5 Hz output
+ exact EgoSpeak split and Table 2 metric
```