icassp2027-experiment/README.md

# Omni-only Turn-Taking Experiment

This project tests whether **Qwen2.5-Omni-3B can replace both the handcrafted
audio/visual feature extractors and the downstream RNN/Transformer used by
EgoSpeak** for egocentric turn-taking anticipation.

The main experiment is deliberately simple:

```text
causal audio + video up to time t
                |
                v
     Qwen2.5-Omni-3B Thinker
                |
        fused state q_t [2048]
                |
                v
       trainable prediction head
                |
                v
  future logits [10 horizons, 3 classes]
```

There is **no RNN after Qwen**. The Omni Thinker is the temporal model. The
only supervised component in the first experiment is a small `10 x 3`
classification head.

## Current status

| File | Status | Purpose |
|---|---|---|
| `smoke_test.py` | Verified on one NVIDIA L20 | Text and 12-second EasyCom audio-video inference |
| `easycom_manifest.py` | Implemented, awaiting dataset-side run | Discover one-minute clips, preserve session timecodes, and check 5 Hz targets |
| `qwen_omni_features.py` | Implemented, not executed here | Load only the 3B Thinker and extract fused states from one AV window |
| `streaming_turn_model.py` | Implemented | Produce one causal Omni state every 200 ms, with causal-prefix and open-stream modes |
| `TurnPredictionHead` | Defined, untrained | Map `[2048]` to `[10, 3]` logits |
| `train_turn_head.py` | Implemented | Train the frozen-Omni anticipation head with the official EasyCom split |
| `infer_turn_head.py` | Implemented | Save EgoSpeak-format predictions and reproduce per-horizon AP/mAP |

The official full model, Thinker input path, embedded audio, video decoding,
and FlashAttention-2 have passed a 12-second EasyCom smoke test on one NVIDIA
L20. Streaming extraction, cache equivalence, target alignment, training, and
paper metrics remain unverified.

## Why Omni-only instead of Omni + RNN

EgoSpeak separately extracts visual and audio features and then uses GRU,
LSTR, or Mamba for temporal modeling:

```text
ResNet50 video feature + wav2vec2 audio feature -> temporal model -> prediction
```

Qwen2.5-Omni already contains:

- audio and visual encoders;
- joint audio-video temporal positions;
- a causal decoder-only Thinker with KV cache;
- pretrained acoustic, visual, and semantic knowledge.

Adding a GRU after a continuously processed Omni sequence would model history
twice and would make it harder to attribute an improvement to Omni. The primary
comparison is therefore:

```text
EgoSpeak: ResNet50 + wav2vec2 + GRU
Ours:     Qwen2.5-Omni Thinker + small prediction head
```

An Omni-feature-plus-GRU model can be kept as an ablation, but it is not the
main architecture.

## EgoSpeak contract that this project reproduces

The inspected reference repository is `/Volumes/760P/Code/EgoSpeak`.
It contains no Qwen or Omni implementation and no raw-media feature extraction
scripts. It distributes precomputed arrays instead:

```text
DATASET/
|- rgb_kinetics_resnet50/   # [T, 2048]
|- audio_wav2vec2/          # [T, 5120]
`- target_perframe/         # [T, 3]
```

The useful part for this project is the exact task interface.

### Time and classes

Both EasyCom and the EgoSpeak `EGO4DSHUFFLE` split use **5 FPS**, so one task
timestep is **0.2 seconds**. The three output columns are:

```text
0: background
1: camera wearer speaking
2: other people speaking
```

They are not strictly one-hot. Background is exclusive, but simultaneous
camera-wearer and other-person speech is stored as `[0, 1, 1]`. The inspected
EasyCom features contain 7,711 such overlap steps. Preserve the full
multi-positive target rather than applying `argmax`.

Relevant EgoSpeak files:

- `LSTR/data/data_info.json`: FPS, classes, and session splits;
- `MiniROAD/configs/EasyCom/miniroad_aa_easycom.yaml`: 128-step training
  windows, stride 4, three classes, and anticipation length 10;
- `MiniROAD/datasets/dataset.py`: feature loading and future-label indexing;
- `MiniROAD/criterions/loss.py`: anticipation loss;
- `MiniROAD/trainer/eval.py`: per-horizon AP and mean anticipation mAP.

### Future target alignment

For current task index `i`, the prediction target is:

```python
target[i + 1 : i + 11]
```

Thus the ten output rows represent:

```text
t+0.2, t+0.4, t+0.6, ..., t+2.0 seconds
```

If `states[i]` is verified to correspond to EgoSpeak task index `i`, the
offline head-training tensors should be constructed as follows:

```python
target = target_perframe                            # [T, 3], multi-positive
usable = min(len(states), len(target)) - 10

x = states[:usable]                                 # [N, 2048]
y = stack(
    [target[h : h + usable] for h in range(1, 11)],
    axis=1,
)                                                   # [N, 10, 3]
```

Do not silently assume that the first saved Omni state matches label index
zero. EgoSpeak does not publish the script that generated its original
ResNet/wav2vec2 features, so the exact zero-time convention is not documented.
Before training, inspect a short clip with known speech boundaries and test the
candidate mappings `states[i] -> target[i]` and `states[i] -> target[i+1]`.
Store the selected offset in every feature file/checkpoint.

### Splits

Use the published split, not a random frame split.

EasyCom:

- train, preserving EgoSpeak's list: `Session_7`, `Session_11`, `Session_10`,
  `Session_6`, `Session_9`, `Session_4`, `Session_8`, `Session_5`, `Session_12`;
- test, preserving EgoSpeak's list: `Session_3`, `Session_2`, `Session_1`.

This is the **EgoSpeak comparison split**. The EasyCom dataset repository
itself recommends a different train/validation/test split; do not substitute
that split when reproducing EgoSpeak Table 2.

Ego4D Audio-Visual Diarization:

- use the `EGO4DSHUFFLE` key;
- 346 training clips;
- 87 test clips;
- each clip is approximately five minutes.

Never mix timesteps from the same session across training and validation.

## EasyCom raw-media layout and manifest

The extracted release used here has this relevant structure:

```text
extracted_easycom/
`- Main/
   |- Video_Compressed/
   |  |- Session_1/
   |  |  |- 00-00-000.mp4
   |  |  |- 01-00-000.mp4
   |  |  `- ...
   |  `- Session_2 ... Session_12/
   |- Glasses_Microphone_Array_Audio/
   |- Close_Microphone_Audio/
   |- Speech_Transcriptions/
   `- Metadata/
```

`Video_Compressed` is the AR-glasses wearer's egocentric stream with embedded
stereo AAC audio. Each session is divided into approximately one-minute clips;
the `mm-ss-SSS` filename is the clip's start time on the original recording
timeline. Some intervals are absent because EasyCom redacted one-minute
segments. These raw time gaps are expected.

EgoSpeak does not preserve those empty intervals. Its appendix states that it
crops every EasyCom clip to retain only the 1920-pixel-wide video region and
then merges the retained clips into one dense video per session. Therefore the
EgoSpeak-compatible label time for a clip is its cumulative position in the
concatenation, not the `mm-ss-SSS` source time. The manifest stores both.

Build a raw-video manifest and, when the EgoSpeak labels are available, check
their duration at 5 Hz:

```bash
python easycom_manifest.py \
  --dataset-root /data1/yfzx/wangx/Datasets/easycom/extracted_easycom \
  --target-root /path/to/EasyCom/target_perframe \
  --output manifests/easycom.json
```

Omit `--target-root` for the first media-only inventory. The manifest reports
raw-source discontinuities as informational and computes each clip's dense
EgoSpeak concatenation offset and corresponding 5 Hz target index. For the
audio-only ablation, use the same embedded glasses audio as A+V (extract it to
mono WAV if needed); do not switch to close microphones, because that changes
the sensor and makes the modality ablation unfair.

Audit downloaded EgoSpeak arrays, and optionally compare every session length
against the raw manifest:

```bash
python audit_egospeak_features.py \
  --feature-root /path/to/EgoSpeak/dataset/EasyCom \
  --raw-manifest manifests/easycom.json \
  --output reports/egospeak_easycom_audit.json
```

The inspected release contains 95,453 synchronized steps across all 12
sessions: 5.302944 hours at 5 Hz. Every session has identical RGB, flow, audio,
and target length. Shapes are `[T,2048]`, `[T,2048]`, `[T,5120]`, and `[T,3]`.

## Recommended comparison matrix

| Experiment | Visual sampling | Audio | Output rate |
|---|---:|---|---:|
| EgoSpeak official A+V | 5 Hz ResNet features | 5 Hz wav2vec2 features | 5 Hz |
| Omni audio-only | none | raw glasses-wearer waveform | 5 Hz |
| Omni A+V primary | 2 FPS | raw embedded waveform | 5 Hz |
| Omni A+V FPS ablation | 5 FPS | raw embedded waveform | 5 Hz |

Two FPS is the primary Omni setting because it matches Qwen's default video
sampling and permits longer context. Five FPS is the visual-rate ablation.
Input feature rates need not match EgoSpeak; the split, causal prediction
times, three classes, ten target horizons, and evaluator must match exactly.

## Prepare an EgoSpeak-compatible EasyCom session

Do not pass the raw one-minute files independently to Omni and do not preserve
the gaps implied by their filenames. EgoSpeak concatenates the retained clips
into a dense session timeline and crops the participant-ID banner on the left.

`prepare_easycom_session.py` reproduces this policy and creates videos whose
stored frames already lie on a fixed session-global 2 FPS or 5 FPS grid. This
prevents Qwen's per-window video loader from choosing a new adaptive set of
frames for every overlapping prefix.

Session 1 has been checked against the local release:

```text
24 retained clips
27,768 source frames at 20 FPS
/ 4 frames per task step
= 6,942 steps at 5 Hz
= target_perframe/Session_1.npy length exactly
```

Preview the commands and validate frames against the target without encoding:

```bash
python prepare_easycom_session.py \
  --session-dir /Volumes/MX500B/Code/easycom/Main/Video_Compressed/Session_1 \
  --output-dir prepared_easycom/Session_1 \
  --target /Volumes/MX500B/Code/EgoSpeak/dataset/EasyCom/target_perframe/Session_1.npy \
  --fps 2 5 \
  --dry-run
```

Remove `--dry-run` to run FFmpeg. The outputs are:

```text
Session_1_2fps.mp4     primary Omni A+V input
Session_1_5fps.mp4     visual-rate ablation
Session_1.json         clip order, exact T, crop and sampling provenance
Session_1.ffconcat     reproducible dense concatenation order
```

The target length must equal the number of complete 5 Hz intervals,
`floor(sum(source_frames) / 4)`. If a session ends with one to three residual
20 Hz frames, the target-derived duration discards only that incomplete final
interval; the discarded frame count is recorded in the sidecar.

Each clip's decoded video and audio are trimmed to the exact duration implied
by its video frame count before concatenation. This prevents AAC priming/tail
padding from accumulating at the 24 clip boundaries and causing progressive
audio/video drift. The session-global visual grid is padded defensively at EOF
and then trimmed by frame index, so it contains exactly
`ceil(target_steps / 5 * video_fps)` frames on every FFmpeg version.

On the remote L20 machine, use its own dataset paths. For example:

```bash
python prepare_easycom_session.py \
  --session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_1 \
  --output-dir /data1/yfzx/wangx/Datasets/easycom/omni_prepared/Session_1 \
  --target /path/to/EgoSpeak/dataset/EasyCom/target_perframe/Session_1.npy \
  --fps 2 5
```

This is the command that performs the potentially long FFmpeg processing; it
has not been run by the code authoring workflow.

If encoding completed but a later validation/metadata step failed, update the
script and reuse the existing MP4 without encoding it again:

```bash
python prepare_easycom_session.py ... --fps 2 --reuse-existing
```

## What Qwen's native 2-second streaming design means

Qwen2.5-Omni was designed for **block-wise multimodal prefill**. Its
[technical report](https://arxiv.org/html/2503.20215) states that the audio
and visual encoders are modified for block-wise processing and that audio
self-attention is performed in independent two-second blocks. The released
[3B configuration](https://huggingface.co/Qwen/Qwen2.5-Omni-3B/raw/main/config.json)
uses `seconds_per_chunk=2`, `position_id_per_seconds=25`, and audio
`n_window=100`.

Conceptually, a native online input is organized as follows:

```text
open one user/media turn
  encode/append AV [0,2]       -> commit Thinker KV
  encode/append AV [2,4]       -> commit Thinker KV
  encode/append AV [4,6]       -> commit Thinker KV
  ...
close the media turn           -> generate text and/or speech
```

Within each temporal group, video tokens precede the aligned audio tokens.
TMRoPE assigns 25 temporal positions per second, or 50 positions per native
two-second group, so audio and video share a physical-time coordinate system.
Completed blocks are appended to one causal Thinker sequence; the long-range
history is modeled by the Thinker rather than by cross-block full attention in
the modality encoders.

This architecture is block-causal, not frame-causal. Attention inside an
audio block is non-causal. It is therefore invalid to encode the complete
`[0,2]` block and claim that its representation at 0.2 seconds saw no future:

```text
encode complete [0,2] AV -> read state at 0.2 s  # future leakage
```

For this project's 5 Hz targets, the current incomplete block must instead be
encoded only through the prediction time. These 0.2-second partial-block
readouts are an extension made for EgoSpeak; the Qwen paper does not define a
canonical hidden state every 200 ms.

The public Transformers and local Qwen web-demo APIs normally receive a
completed media object and call the processor/model once. They retain the
model's block-wise masks and token organization, but do not expose the full
persistent live-session implementation used by the hosted real-time product.

## Native Qwen streaming versus `causal-prefix`

The two extraction modes are both causal with respect to raw observations,
but they define different model inputs. `causal-prefix` is a sliding-window
reference invented for this experiment; it is not Qwen's native streaming
execution model.

| Property | `causal-prefix` | Native Qwen streaming idea |
|---|---|---|
| Session | New inference at every 0.2 s | One open media turn |
| Media organization | One complete `[max(0,t-M),t]` item | Fixed source-aligned 2 s blocks |
| Modality encoding | Re-encode the complete window | Encode/commit each completed block once |
| Thinker KV | Start empty for every state | Append to persistent KV |
| TMRoPE | Recomputed from each window origin | Advances with the stream |
| History | Exact rolling `M` seconds | Accumulates until a serving/context policy intervenes |
| 5 Hz readout | Close/read a new turn each time | Not specified by the original model |

The difference becomes especially important after a sliding window fills. At
`t=20.8` with `M=20`, `causal-prefix` constructs one new media item covering:

```text
[0.8,20.8]
  -> encoder blocks [0.8,2.8], [2.8,4.8], ..., [18.8,20.8]
```

At `t=21.0`, every encoder boundary moves again:

```text
[1.0,21.0]
  -> encoder blocks [1.0,3.0], [3.0,5.0], ..., [19.0,21.0]
```

Native streaming keeps the source-aligned partition instead:

```text
[0,2], [2,4], ..., [18,20], current partial [20,20.8]
```

Because Qwen's audio encoder uses block-local non-causal attention, moving a
block boundary changes the representation even when most raw samples are the
same. Window-relative TMRoPE, convolution/padding boundaries, chat closure,
and BF16/attention execution add further differences. This is why exact KV
replay can be numerically identical to its own from-scratch replay while still
differing from `causal-prefix`; the modes are evaluating different token and
encoder histories.

Use the modes according to the scientific question:

- use `open-stream-replay` for the primary online experiment, training, and
  deployment-aligned evaluation;
- use `causal-prefix` as an expensive continuous-window reference and
  short-clip ablation;
- never train a head on one representation and deploy it on the other without
  retraining, because their feature distributions are not interchangeable.

## Causal 5 Hz extraction modes

`streaming_turn_model.py` implements two modes.

### 1. `causal-prefix`: continuous-window reference

At every 200 ms, independently process the causal context ending at the
current time:

```text
[max(0, t-20 s), t] AV -> Omni -> q_t
```

Properties:

- strict no-future input;
- one continuous media window per prediction;
- no cross-forward cache;
- very expensive because overlapping context is recomputed;
- useful for checking continuous sliding-window semantics and causality;
- not the native Qwen block-streaming execution model.

Example:

```bash
python streaming_turn_model.py \
  --video /path/to/easycom_session.mp4 \
  --use-audio-in-video \
  --mode causal-prefix \
  --start 0 \
  --end 20 \
  --memory-seconds 20 \
  --output features/easycom_20s_prefix.pt
```

### 2. `open-stream-replay`: recommended efficient mode

This mode keeps one user/media turn open across the whole bounded-memory
segment. The first block contributes the chat and AV opening markers once;
later completed two-second blocks contribute only interleaved audio/video
payload tokens:

```text
open AV prefix
  completed [0,2] payload       -> commit KV
  current   [2,2.2] payload     -> temporary branch -> close/read q_t
  current   [2,2.4] payload     -> temporary branch -> close/read q_t
  ...
  completed [2,4] payload       -> commit KV
AV/chat closure                 -> never committed; readout branch only
```

This matches the core Qwen design choices: fixed two-second AV groups, one
open media turn, completed-block commits, and incremental Thinker KV. It makes
two practical changes for this dataset:

1. the current 0--2 second partial block is evaluated speculatively every
   0.2 seconds and discarded unless it reaches a complete block boundary;
2. the KV history is bounded and rebuilt from recent complete blocks rather
   than accumulating for an entire multi-minute session.

It should therefore be described as **bounded Qwen-style open streaming**, not
as bit-for-bit reproduction of an unbounded official serving session.

The audio encoder accepts a shorter final chunk with padding and an
actual-length mask. For the two-frame visual patch requirement, strict-grid
mode duplicates only the most recent past frame when necessary.

Run a four-second validation before using it for extraction:

```bash
python streaming_turn_model.py \
  --model-id /path/to/Qwen2.5-Omni-3B \
  --video prepared_easycom/Session_1/Session_1_2fps.mp4 \
  --use-audio-in-video \
  --mode open-stream-replay \
  --start 0 --end 4 \
  --video-fps 2 \
  --strict-fixed-video-grid \
  --verify-every 1 \
  --verify-continuous-every 1 \
  --flash-attention-2 \
  --output features/Session_1_open_first_4s.pt
```

`cache_replay` must be identical to replaying the same open stream from an
empty cache. `continuous_window_semantics` compares it with one normal
continuous Qwen AV input. Do not assume equivalence: the modality encoders are
still invoked per tail block in the efficient path, so this numerical check is
required even though artificial user-turn boundaries have been removed.

## Whole-session prediction and the 20-second limit

Twenty seconds is a practical memory/context setting for the 3090, not the
maximum video duration and not a limit specified by Qwen. The original
append-only idea eventually reaches both linear KV-memory growth and the
Thinker's 32,768-token context limit, so an unbounded multi-minute stream is
not practical. The complete session is processed sequentially and a
prediction is emitted every 0.2 seconds.

`open-stream-replay` uses blockwise rolling memory. With a 20-second
budget it rebuilds the open cache from the latest nine completed blocks
(18 seconds) when a new block starts, then adds the current 0--2 second tail.
Its effective history therefore varies from 18 to 20 seconds and never drops
to an empty 0.2-second context at a 20-second boundary.

For a short native-stream fidelity check, set `--memory-seconds` to at least
the tested clip duration. For example, the options
`--start 0 --end 40 --memory-seconds 40` prevent a rolling rebuild during that
40-second clip.
Do not extend this policy to complete EasyCom/Ego4D sessions: KV memory grows
linearly, and this implementation clones the cache for speculative partial
blocks and readout branches.

For the paper experiment, drive the output length from the preparation sidecar
rather than a floating MP4 duration:

```bash
python streaming_turn_model.py \
  --session-metadata prepared_easycom/Session_1/Session_1.json \
  --use-audio-in-video \
  --mode open-stream-replay \
  --memory-seconds 20 \
  --video-fps 2 \
  --strict-fixed-video-grid \
  --verify-every 25 \
  --verify-continuous-every 25 \
  --output features/Session_1_omni_2fps.pt
```

The sidecar selects the matching fixed-FPS MP4 and sets `N` to its exact
`num_steps`. Alternatively pass `--video ... --target ...` explicitly. A raw
20 FPS MP4 with `--video-fps 2` is rejected when
`--strict-fixed-video-grid` is enabled.

In strict A+V mode the extractor does not delegate temporal frame selection to
the generic video utility. It reads the already-fixed MP4 grid explicitly and
assigns both video frames and audio samples with half-open intervals
`[start, end)`. Thus a frame exactly on a two-second boundary belongs only to
the following block. It duplicates the most recent available frame when the
visual tower needs an even frame count. Audio is loaded once at 16 kHz and
sliced at the same causal boundary. The processor still receives
`use_audio_in_video=True`, so Qwen's audio/video token interleaving is retained.
Long runs use `tqdm` and update the progress bar after every completed state,
showing percentage, media time, elapsed time, ETA, and throughput. Pass
`--no-progress` to disable it. The Thinker-only path also suppresses Qwen's
irrelevant speech-output system prompt warning.

Before a full session, run only the first four seconds and inspect both check
types:

```bash
python streaming_turn_model.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --video /data1/yfzx/wangx/Datasets/easycom/omni_prepared/Session_1/Session_1_2fps.mp4 \
  --use-audio-in-video \
  --mode open-stream-replay \
  --start 0 --end 4 \
  --video-fps 2 \
  --strict-fixed-video-grid \
  --verify-every 1 \
  --verify-continuous-every 1 \
  --flash-attention-2 \
  --output features/Session_1_first_4s_check.pt
```

Do not launch all sessions unless `cache_replay` cosine is nearly 1. For
`open-stream-replay`, a materially lower `continuous_window_semantics` cosine
means that the custom payload stitching is not equivalent to one continuous
Qwen input; it must not be dismissed as a media-turn-boundary effect because
this mode intentionally keeps a single media turn open.

Video can be sampled at 2 FPS while predictions remain at 5 Hz. Audio and the
causal boundary update every 200 ms; consecutive prediction times may use the
same most-recent visual frame. A frame with timestamp later than `t` must never
be included.

## Direct extraction without a prepared 2 FPS MP4

`streaming_turn_model_directly.py` removes the offline video-transcoding
requirement. It reads original frame PTS with Decord, constructs a global
2 FPS lattice, maps every lattice point to the latest source frame whose PTS
is not later than that point, and passes explicit frame lists to Qwen. Audio is
decoded once to 16-kHz mono and sliced at exact causal boundaries. Therefore
no FFmpeg `fps` filter or second H.264/AAC encode can alter the experiment.

Validate one Ego4D clip without loading Qwen:

```bash
python streaming_turn_model_directly.py \
  --video /data/wangx/Datasets/ego4d/v1/clips/CLIP_UID.mp4 \
  --target /data/wangx/Datasets/ego4d/ego4d_target_frame/CLIP_UID.npy \
  --video-fps 2 \
  --media-only
```

Run a short native persistent-KV check:

```bash
python streaming_turn_model_directly.py \
  --model-id /data/wangx/Models/Qwen2.5-Omni-3B \
  --video /data/wangx/Datasets/ego4d/v1/clips/CLIP_UID.mp4 \
  --end 40 \
  --mode open-stream-replay \
  --memory-seconds 0 \
  --video-fps 2 \
  --flash-attention-2 \
  --device cuda:0 \
  --output features/direct/CLIP_UID_first_40s.pt
```

Here `--memory-seconds 0` means that completed-block Thinker KV persists for
the entire requested interval. Do not use this setting blindly for complete
multi-minute recordings: it has linear KV growth and remains subject to the
Thinker's context limit. A full bounded-memory dataset run should explicitly
use the experimental budget, for example `--memory-seconds 20`.

EasyCom can be read directly as the dense concatenation of retained clips;
filename timecode gaps are ignored and the rightmost 1920x1080 region is kept:

```bash
python streaming_turn_model_directly.py \
  --model-id /path/to/Qwen2.5-Omni-3B \
  --easycom-session-dir /path/to/Main/Video_Compressed/Session_1 \
  --target target_frame/Session_1.npy \
  --mode open-stream-replay \
  --memory-seconds 20 \
  --video-fps 2 \
  --flash-attention-2 \
  --device cuda:0 \
  --output features/direct/Session_1.pt
```

The direct reader defaults to Decord. On NVIDIA systems, install
`PyNvVideoCodec` and select the NVDEC backend explicitly:

```bash
python -m pip install PyNvVideoCodec

CUDA_VISIBLE_DEVICES=0 python streaming_turn_model_directly.py \
  --model-id /path/to/Qwen2.5-Omni-3B \
  --easycom-session-dir /path/to/Main/Video_Compressed/Session_1 \
  --end 24 \
  --mode open-stream-replay \
  --memory-seconds 20 \
  --video-fps 2 \
  --frame-cache-size 64 \
  --decoder pynvcodec \
  --decoder-gpu-id 0 \
  --flash-attention-2 \
  --device cuda:0 \
  --output features/direct/Session_1_24s_pynvcodec.pt
```

The NVDEC backend uses ffprobe for exact display-order PTS indexing, one
reconfigurable `SimpleDecoder` for pixel decode, and the existing ffmpeg PCM
audio path. `--decoder-gpu-id` is relative to `CUDA_VISIBLE_DEVICES`, so it is
normally `0` even when the worker is assigned a nonzero host GPU. Decoder RGB
conversion can differ slightly; validate a short Decord/PyNvVideoCodec pair
before extracting every session, and do not mix decoder backends across
train/test splits.

Matching `states.shape[0]` to the target is necessary but not sufficient.
The direct reader also validates EasyCom's 20-Hz source inventory, source PTS
monotonicity, dense duration, crop bounds, audio sample count, and causal
frame-to-grid mapping. The original prepare scripts remain useful only as an
optional decode-speed/reproducibility cache.

## Saved streaming tensors

Each `.pt` file contains:

- `times`: `[N]`, prediction times in seconds;
- `states`: `[N, 2048]`, causal fused Thinker states;
- `label_indices`: `[0, ..., N-1]`, direct `target_perframe` row alignment;
- `valid_10_horizon_steps`: `N-10`, because the last ten current rows do not
  have all future labels;
- `mode`: `causal-prefix` or `open-stream-replay`;
- `stream_representation`: explicit input-organization metadata;
- `config`: step, native block, memory, FPS, and pixel settings;
- `verification`: cached-versus-exact numerical checks;
- `input_video_fps` and `video_sampling_grid`: sampling provenance;
- `media_interval_policy`: fixed-grid AV uses half-open `[start, end)` ownership;
- `logits`: optional `[N, 10, 3]` output if a trained head is supplied;
- `probabilities`: optional softmax probabilities.

## Training the `10 x 3` head

The head must be trained. It is not part of pretrained Qwen.

The first experiment should freeze all Qwen parameters and train only:

```text
LayerNorm(2048)
-> Linear(2048, 1024)
-> GELU
-> Dropout
-> Linear(1024, 30)
-> reshape [10, 3]
```

`easycom_head.py` defines the head and loss; they remain re-exported by
`streaming_turn_model.py` for compatibility:

```python
from streaming_turn_model import (
    TurnPredictionHead,
    egospeak_multihorizon_loss,
)
```

The supplied LSTR config uses Adam, 50 epochs, learning rate `7e-5`, weight
decay `5e-5`, dropout `0.2`, and 10 warmup epochs followed by cosine decay.
`train_turn_head.py` uses those defaults. Its batch size is 512 causal states,
not LSTR's 16 temporal windows, because the two model inputs have different
shapes.

Train once on the official nine sessions with a fixed 50-epoch protocol:

```bash
python train_turn_head.py \
  --feature-root features/open_2fps \
  --target-root target_frame \
  --output checkpoints/open_2fps_head.pth \
  --device cuda:0
```

The default loss is `lstr-mce`, exactly matching LSTR's MCE treatment of a
multi-positive `[0,1,1]` row: it sums both active negative-log-softmax terms
without normalizing the target. The previous L2-normalized objective belongs
to MiniROAD and is available only as the explicit
`--loss miniroad-normalized` ablation.

Recommended experimental procedure:

1. Freeze Qwen and extract all training-session states offline.
2. Verify state/label time offset and truncate only the final ten invalid
   current positions.
3. Keep EgoSpeak targets as float multi-positive tensors `[N, 10, 3]`; never
   convert them with `argmax`.
4. For comparison with the supplied LSTR shell, train with the default
   `egospeak_multihorizon_loss`/`lstr-mce` objective.
5. Select hyperparameters using only training sessions. For EasyCom, reserve a
   training session for development or use session-level cross-validation;
   never tune on Sessions 1--3.
6. After choosing the epoch/hyperparameters, optionally retrain on all nine
   EasyCom training sessions and evaluate the fixed model once on the test set.

Class-balanced/focal loss, a learned turn-query token, and LoRA are later
ablations. They should not replace the frozen-Omni linear/MLP-head baseline.

## EgoSpeak-compatible evaluation

Run inference on the official test sessions (`Session_3`, `Session_2`, and
`Session_1`):

```bash
python infer_turn_head.py \
  --feature-root features/open_2fps \
  --target-root target_frame \
  --checkpoint checkpoints/open_2fps_head.pth \
  --output-dir results/open_2fps_head \
  --device cuda:0
```

This writes `pred/Session_X.npy` with the exact LSTR/EgoSpeak layout
`[target_time, class, future_horizon]`, i.e. `[T,3,10]`, plus `metrics.json`.
For every valid current timestep, the head itself produces probabilities
shaped:

```text
[number of timesteps, 10 future horizons, 3 classes]
```

Then reproduce EgoSpeak's Table 2 procedure:

1. for each horizon independently, concatenate test-session predictions and
   ground truth;
2. compute per-class frame-level average precision;
3. average the three class AP values to obtain horizon mAP;
4. report all ten horizon mAP values;
5. average the ten horizon mAP values for the final score.

With EgoSpeak's `ignore_index=-100`, the released evaluator does not remove
class index zero, so background is included. The script reports two protocols:

- `egospeak_full_timeline`: the same full `[T,3,10]` array and evaluator as
  LSTR, with zeros where a prediction would require pre-session context;
- `causal_valid_only`: excludes only those unavailable first `h` rows for
  horizon `h`. This is the primary strictly causal score.

The difference involves at most the first ten rows of each session, but both
are saved so the comparison is explicit.

The convenience wrapper mirrors EgoSpeak's train/inference shells:

```bash
chmod +x run_easycom_head_experiment.sh
CUDA_VISIBLE_DEVICES=0 ./run_easycom_head_experiment.sh all
```

Reference Table 2 targets from the EgoSpeak paper:

- EasyCom GRU A+V: **60.6 mean mAP**;
- EasyCom Transformer A+V: **58.7 mean mAP**;
- Ego4D Transformer audio: **69.2 mean mAP**;
- Ego4D Transformer A+V: **69.0 mean mAP**;
- Ego4D GRU audio: **69.2 mean mAP**;
- Ego4D GRU A+V: **68.2 mean mAP**.

Do not compare these numbers with EgoSpeak Table 3 target-speaker-only AP;
Table 3 uses a different metric.

## Experiments that demonstrate Omni's semantic advantage

The headline comparison alone cannot establish that improvements come from
semantics. Run the following controlled experiments:

1. **Omni audio vs Omni audio+video**: measure whether visual grounding helps.
2. **Per-horizon curve**: semantic completion should be especially visible at
   longer horizons such as 1.2--2.0 seconds.
3. **Context length**: compare 4, 10, 15, and 20 seconds without adding an RNN.
4. **Layer ablation**: compare audio-encoder output and several Thinker layers,
   because an intermediate layer may be more discriminative than the final
   generation-oriented layer.
5. **Semantic-completion subset**: contrast completed utterances with
   hesitation/incomplete pauses while matching pause duration and basic
   prosody as closely as possible.
6. **Question/statement/backchannel subsets**: report target-speaker AP and
   premature-interruption rate on linguistically meaningful events.
7. **Causal transcript baseline**: compare Omni audio with causal ASR text plus
   a language model to isolate lexical/semantic contribution.

The frozen-Omni head is the cleanest representation test. LoRA can be reported
as a second-stage ceiling, not as the only result.

## Single-window feature inspection

`qwen_omni_features.py` is useful for inspecting a short window or layer, but
it is not the main whole-session experiment.

If audio is embedded in the video:

```bash
python qwen_omni_features.py \
  --video /path/to/window.mp4 \
  --use-audio-in-video \
  --output features/window.pt
```

If audio and video are separate:

```bash
python qwen_omni_features.py \
  --video /path/to/window.mp4 \
  --audio /path/to/window.wav \
  --output features/window.pt
```

Do not provide the same audio both as a separate file and via
`--use-audio-in-video`.

Saved single-window fields are:

- `token_features`: `[B, token_length, 2048]`;
- `last_token`: `[B, 2048]`;
- `tail_mean`: `[B, 2048]`;
- `attention_mask`: valid-token mask.

## Installation

Install `ffmpeg`, then:

```bash
pip install -r requirements.txt
```

The implementation is pinned to `transformers==4.52.3` because the
open-stream replay path uses version-specific Thinker internals. On Linux,
`qwen-omni-utils[decord]` is recommended for faster video decoding. On a
compatible NVIDIA GPU, optionally install `flash-attn` and pass
`--flash-attention-2`.

## Ego4D Audio-Visual Diarization download

Do not download the complete Ego4D release for this experiment. Download only
the compressed Audio-Visual Diarization benchmark clips and annotations:

```bash
ego4d \
  --output_directory=/path/to/ego4d \
  --datasets clips annotations \
  --benchmarks AV
```

The approximately 1 TB figure on the Ego4D download page refers to all
benchmark clips combined, not only the AV subset. The annotated AVD subset has
about 572 five-minute clips (roughly 50 hours). EgoSpeak uses a custom 346/87
train/test split, about 36.1 hours total. Reproducing its Table 2 requires the
exact IDs in EgoSpeak's `EGO4DSHUFFLE` split.

## Required validation before a full run

Do not launch full EasyCom/Ego4D extraction until all of these pass:

- no input audio sample or video frame has timestamp later than prediction
  time `t`;
- `states` and `target_perframe` lengths/time origins are reconciled and the
  chosen label offset is saved;
- open-stream cache replay is exactly equivalent to its from-scratch replay on
  short clips;
- open-stream and causal-prefix predictions are compared on identical clips;
- a 20-second run fits the 3090 with the selected pixel budget and attention
  implementation;
- saved outputs contain exactly ten future horizons in the order 0.2--2.0 s;
- the local metric reproduces EgoSpeak's evaluator on known predictions;
- test session IDs never enter training, validation, normalization statistics,
  or hyperparameter selection.

Once these checks pass, the first publishable experiment is:

```text
frozen Qwen2.5-Omni-3B Thinker
+ trained 10x3 head
+ bounded Qwen-style 2-second open streaming
+ 20-second rolling causal memory
+ full-session 5 Hz output
+ exact EgoSpeak split and Table 2 metric
```