Add scripts for preparing and extracting 5 FPS audio-video clips, including prepare_ego4d_2fps.py, run_extract_5fps_2pergpu.sh, and run_prepare_all_5fps.sh. Introduce streaming_turn_model_directly.py for direct extraction from original media. Update README.md to reflect changes in Qwen's streaming design and causal extraction modes. Add new AGENTS.md file for documentation.
This commit is contained in:
Xin Wang
@@ -317,33 +317,112 @@ script and reuse the existing MP4 without encoding it again:
|
||||
python prepare_easycom_session.py ... --fps 2 --reuse-existing
|
||||
```
|
||||
|
||||
## What Qwen's native 2-second chunk means
|
||||
## What Qwen's native 2-second streaming design means
|
||||
|
||||
Qwen2.5-Omni is **not restricted to receiving audio once every two seconds**.
|
||||
It can receive an arbitrary-duration AV input. The two-second value describes
|
||||
internal processing:
|
||||
Qwen2.5-Omni was designed for **block-wise multimodal prefill**. Its
|
||||
[technical report](https://arxiv.org/html/2503.20215) states that the audio
|
||||
and visual encoders are modified for block-wise processing and that audio
|
||||
self-attention is performed in independent two-second blocks. The released
|
||||
[3B configuration](https://huggingface.co/Qwen/Qwen2.5-Omni-3B/raw/main/config.json)
|
||||
uses `seconds_per_chunk=2`, `position_id_per_seconds=25`, and audio
|
||||
`n_window=100`.
|
||||
|
||||
1. the audio encoder divides acoustic features into approximately two-second
|
||||
attention blocks;
|
||||
2. audio and video tokens are interleaved in two-second temporal groups before
|
||||
entering the Thinker.
|
||||
|
||||
The audio encoder attention inside a block is non-causal. Therefore this is
|
||||
invalid for online evaluation:
|
||||
Conceptually, a native online input is organized as follows:
|
||||
|
||||
```text
|
||||
encode the complete [0, 2.0 s] block
|
||||
-> read a purported state at 0.2 s
|
||||
open one user/media turn
|
||||
encode/append AV [0,2] -> commit Thinker KV
|
||||
encode/append AV [2,4] -> commit Thinker KV
|
||||
encode/append AV [4,6] -> commit Thinker KV
|
||||
...
|
||||
close the media turn -> generate text and/or speech
|
||||
```
|
||||
|
||||
That early state can contain audio from 0.2--2.0 seconds. At prediction time
|
||||
`t`, the model must receive only audio and video with timestamps `<= t`.
|
||||
Within each temporal group, video tokens precede the aligned audio tokens.
|
||||
TMRoPE assigns 25 temporal positions per second, or 50 positions per native
|
||||
two-second group, so audio and video share a physical-time coordinate system.
|
||||
Completed blocks are appended to one causal Thinker sequence; the long-range
|
||||
history is modeled by the Thinker rather than by cross-block full attention in
|
||||
the modality encoders.
|
||||
|
||||
This architecture is block-causal, not frame-causal. Attention inside an
|
||||
audio block is non-causal. It is therefore invalid to encode the complete
|
||||
`[0,2]` block and claim that its representation at 0.2 seconds saw no future:
|
||||
|
||||
```text
|
||||
encode complete [0,2] AV -> read state at 0.2 s # future leakage
|
||||
```
|
||||
|
||||
For this project's 5 Hz targets, the current incomplete block must instead be
|
||||
encoded only through the prediction time. These 0.2-second partial-block
|
||||
readouts are an extension made for EgoSpeak; the Qwen paper does not define a
|
||||
canonical hidden state every 200 ms.
|
||||
|
||||
The public Transformers and local Qwen web-demo APIs normally receive a
|
||||
completed media object and call the processor/model once. They retain the
|
||||
model's block-wise masks and token organization, but do not expose the full
|
||||
persistent live-session implementation used by the hosted real-time product.
|
||||
|
||||
## Native Qwen streaming versus `causal-prefix`
|
||||
|
||||
The two extraction modes are both causal with respect to raw observations,
|
||||
but they define different model inputs. `causal-prefix` is a sliding-window
|
||||
reference invented for this experiment; it is not Qwen's native streaming
|
||||
execution model.
|
||||
|
||||
| Property | `causal-prefix` | Native Qwen streaming idea |
|
||||
|---|---|---|
|
||||
| Session | New inference at every 0.2 s | One open media turn |
|
||||
| Media organization | One complete `[max(0,t-M),t]` item | Fixed source-aligned 2 s blocks |
|
||||
| Modality encoding | Re-encode the complete window | Encode/commit each completed block once |
|
||||
| Thinker KV | Start empty for every state | Append to persistent KV |
|
||||
| TMRoPE | Recomputed from each window origin | Advances with the stream |
|
||||
| History | Exact rolling `M` seconds | Accumulates until a serving/context policy intervenes |
|
||||
| 5 Hz readout | Close/read a new turn each time | Not specified by the original model |
|
||||
|
||||
The difference becomes especially important after a sliding window fills. At
|
||||
`t=20.8` with `M=20`, `causal-prefix` constructs one new media item covering:
|
||||
|
||||
```text
|
||||
[0.8,20.8]
|
||||
-> encoder blocks [0.8,2.8], [2.8,4.8], ..., [18.8,20.8]
|
||||
```
|
||||
|
||||
At `t=21.0`, every encoder boundary moves again:
|
||||
|
||||
```text
|
||||
[1.0,21.0]
|
||||
-> encoder blocks [1.0,3.0], [3.0,5.0], ..., [19.0,21.0]
|
||||
```
|
||||
|
||||
Native streaming keeps the source-aligned partition instead:
|
||||
|
||||
```text
|
||||
[0,2], [2,4], ..., [18,20], current partial [20,20.8]
|
||||
```
|
||||
|
||||
Because Qwen's audio encoder uses block-local non-causal attention, moving a
|
||||
block boundary changes the representation even when most raw samples are the
|
||||
same. Window-relative TMRoPE, convolution/padding boundaries, chat closure,
|
||||
and BF16/attention execution add further differences. This is why exact KV
|
||||
replay can be numerically identical to its own from-scratch replay while still
|
||||
differing from `causal-prefix`; the modes are evaluating different token and
|
||||
encoder histories.
|
||||
|
||||
Use the modes according to the scientific question:
|
||||
|
||||
- use `open-stream-replay` for the primary online experiment, training, and
|
||||
deployment-aligned evaluation;
|
||||
- use `causal-prefix` as an expensive continuous-window reference and
|
||||
short-clip ablation;
|
||||
- never train a head on one representation and deploy it on the other without
|
||||
retraining, because their feature distributions are not interchangeable.
|
||||
|
||||
## Causal 5 Hz extraction modes
|
||||
|
||||
`streaming_turn_model.py` implements two modes.
|
||||
|
||||
### 1. `causal-prefix`: correctness and semantic baseline
|
||||
### 1. `causal-prefix`: continuous-window reference
|
||||
|
||||
At every 200 ms, independently process the causal context ending at the
|
||||
current time:
|
||||
@@ -358,7 +437,8 @@ Properties:
|
||||
- one continuous media window per prediction;
|
||||
- no cross-forward cache;
|
||||
- very expensive because overlapping context is recomputed;
|
||||
- preferred reference when checking semantics and causality.
|
||||
- useful for checking continuous sliding-window semantics and causality;
|
||||
- not the native Qwen block-streaming execution model.
|
||||
|
||||
Example:
|
||||
|
||||
@@ -390,6 +470,18 @@ open AV prefix
|
||||
AV/chat closure -> never committed; readout branch only
|
||||
```
|
||||
|
||||
This matches the core Qwen design choices: fixed two-second AV groups, one
|
||||
open media turn, completed-block commits, and incremental Thinker KV. It makes
|
||||
two practical changes for this dataset:
|
||||
|
||||
1. the current 0--2 second partial block is evaluated speculatively every
|
||||
0.2 seconds and discarded unless it reaches a complete block boundary;
|
||||
2. the KV history is bounded and rebuilt from recent complete blocks rather
|
||||
than accumulating for an entire multi-minute session.
|
||||
|
||||
It should therefore be described as **bounded Qwen-style open streaming**, not
|
||||
as bit-for-bit reproduction of an unbounded official serving session.
|
||||
|
||||
The audio encoder accepts a shorter final chunk with padding and an
|
||||
actual-length mask. For the two-frame visual patch requirement, strict-grid
|
||||
mode duplicates only the most recent past frame when necessary.
|
||||
@@ -420,15 +512,26 @@ required even though artificial user-turn boundaries have been removed.
|
||||
## Whole-session prediction and the 20-second limit
|
||||
|
||||
Twenty seconds is a practical memory/context setting for the 3090, not the
|
||||
maximum video duration. The complete session is processed sequentially and a
|
||||
maximum video duration and not a limit specified by Qwen. The original
|
||||
append-only idea eventually reaches both linear KV-memory growth and the
|
||||
Thinker's 32,768-token context limit, so an unbounded multi-minute stream is
|
||||
not practical. The complete session is processed sequentially and a
|
||||
prediction is emitted every 0.2 seconds.
|
||||
|
||||
`open-stream-replay` instead uses blockwise rolling memory. With a 20-second
|
||||
`open-stream-replay` uses blockwise rolling memory. With a 20-second
|
||||
budget it rebuilds the open cache from the latest nine completed blocks
|
||||
(18 seconds) when a new block starts, then adds the current 0--2 second tail.
|
||||
Its effective history therefore varies from 18 to 20 seconds and never drops
|
||||
to an empty 0.2-second context at a 20-second boundary.
|
||||
|
||||
For a short native-stream fidelity check, set `--memory-seconds` to at least
|
||||
the tested clip duration. For example, the options
|
||||
`--start 0 --end 40 --memory-seconds 40` prevent a rolling rebuild during that
|
||||
40-second clip.
|
||||
Do not extend this policy to complete EasyCom/Ego4D sessions: KV memory grows
|
||||
linearly, and this implementation clones the cache for speculative partial
|
||||
blocks and readout branches.
|
||||
|
||||
For the paper experiment, drive the output length from the preparation sidecar
|
||||
rather than a floating MP4 duration:
|
||||
|
||||
@@ -488,6 +591,68 @@ causal boundary update every 200 ms; consecutive prediction times may use the
|
||||
same most-recent visual frame. A frame with timestamp later than `t` must never
|
||||
be included.
|
||||
|
||||
## Direct extraction without a prepared 2 FPS MP4
|
||||
|
||||
`streaming_turn_model_directly.py` removes the offline video-transcoding
|
||||
requirement. It reads original frame PTS with Decord, constructs a global
|
||||
2 FPS lattice, maps every lattice point to the latest source frame whose PTS
|
||||
is not later than that point, and passes explicit frame lists to Qwen. Audio is
|
||||
decoded once to 16-kHz mono and sliced at exact causal boundaries. Therefore
|
||||
no FFmpeg `fps` filter or second H.264/AAC encode can alter the experiment.
|
||||
|
||||
Validate one Ego4D clip without loading Qwen:
|
||||
|
||||
```bash
|
||||
python streaming_turn_model_directly.py \
|
||||
--video /data/wangx/Datasets/ego4d/v1/clips/CLIP_UID.mp4 \
|
||||
--target /data/wangx/Datasets/ego4d/ego4d_target_frame/CLIP_UID.npy \
|
||||
--video-fps 2 \
|
||||
--media-only
|
||||
```
|
||||
|
||||
Run a short native persistent-KV check:
|
||||
|
||||
```bash
|
||||
python streaming_turn_model_directly.py \
|
||||
--model-id /data/wangx/Models/Qwen2.5-Omni-3B \
|
||||
--video /data/wangx/Datasets/ego4d/v1/clips/CLIP_UID.mp4 \
|
||||
--end 40 \
|
||||
--mode open-stream-replay \
|
||||
--memory-seconds 0 \
|
||||
--video-fps 2 \
|
||||
--flash-attention-2 \
|
||||
--device cuda:0 \
|
||||
--output features/direct/CLIP_UID_first_40s.pt
|
||||
```
|
||||
|
||||
Here `--memory-seconds 0` means that completed-block Thinker KV persists for
|
||||
the entire requested interval. Do not use this setting blindly for complete
|
||||
multi-minute recordings: it has linear KV growth and remains subject to the
|
||||
Thinker's context limit. A full bounded-memory dataset run should explicitly
|
||||
use the experimental budget, for example `--memory-seconds 20`.
|
||||
|
||||
EasyCom can be read directly as the dense concatenation of retained clips;
|
||||
filename timecode gaps are ignored and the rightmost 1920x1080 region is kept:
|
||||
|
||||
```bash
|
||||
python streaming_turn_model_directly.py \
|
||||
--model-id /path/to/Qwen2.5-Omni-3B \
|
||||
--easycom-session-dir /path/to/Main/Video_Compressed/Session_1 \
|
||||
--target target_frame/Session_1.npy \
|
||||
--mode open-stream-replay \
|
||||
--memory-seconds 20 \
|
||||
--video-fps 2 \
|
||||
--flash-attention-2 \
|
||||
--device cuda:0 \
|
||||
--output features/direct/Session_1.pt
|
||||
```
|
||||
|
||||
Matching `states.shape[0]` to the target is necessary but not sufficient.
|
||||
The direct reader also validates EasyCom's 20-Hz source inventory, source PTS
|
||||
monotonicity, dense duration, crop bounds, audio sample count, and causal
|
||||
frame-to-grid mapping. The original prepare scripts remain useful only as an
|
||||
optional decode-speed/reproducibility cache.
|
||||
|
||||
## Saved streaming tensors
|
||||
|
||||
Each `.pt` file contains:
|
||||
@@ -540,9 +705,9 @@ Train once on the official nine sessions with a fixed 50-epoch protocol:
|
||||
|
||||
```bash
|
||||
python train_turn_head.py \
|
||||
--feature-root features/causal_prefix_2fps \
|
||||
--feature-root features/open_2fps \
|
||||
--target-root target_frame \
|
||||
--output checkpoints/causal_prefix_2fps_head.pth \
|
||||
--output checkpoints/open_2fps_head.pth \
|
||||
--device cuda:0
|
||||
```
|
||||
|
||||
@@ -577,10 +742,10 @@ Run inference on the official test sessions (`Session_3`, `Session_2`, and
|
||||
|
||||
```bash
|
||||
python infer_turn_head.py \
|
||||
--feature-root features/causal_prefix_2fps \
|
||||
--feature-root features/open_2fps \
|
||||
--target-root target_frame \
|
||||
--checkpoint checkpoints/causal_prefix_2fps_head.pth \
|
||||
--output-dir results/causal_prefix_2fps_head \
|
||||
--checkpoint checkpoints/open_2fps_head.pth \
|
||||
--output-dir results/open_2fps_head \
|
||||
--device cuda:0
|
||||
```
|
||||
|
||||
@@ -743,7 +908,8 @@ Once these checks pass, the first publishable experiment is:
|
||||
```text
|
||||
frozen Qwen2.5-Omni-3B Thinker
|
||||
+ trained 10x3 head
|
||||
+ 20-second causal memory
|
||||
+ bounded Qwen-style 2-second open streaming
|
||||
+ 20-second rolling causal memory
|
||||
+ full-session 5 Hz output
|
||||
+ exact EgoSpeak split and Table 2 metric
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user