Speaker-Grounded Omni Floor-State Forecasting

This project studies whether a strictly causal, end-to-end pretrained Qwen2.5-Omni-3B backbone contains speaker-grounded audiovisual evidence, and whether that evidence can improve the floor-entry and floor-yield actions of an invisible egocentric wearer in natural multiparty conversations.

Replacing EgoSpeak's handcrafted encoders and downstream GRU/Transformer is retained as a Stage-0 representation baseline. It is no longer the final paper question. The central test is behavioral: at matched wearer-onset recall, correct person-aligned vision should reduce false wearer entries while two other people are conversing, without reducing responses addressed to the wearer or degrading wearer yield decisions.

Repository navigation

Start with PROJECT_CONTEXT.md for the concise code map, data contracts, current blocker and canonical pipelines. This README is the long operational guide and experiment history.

Location	Contents
AGENTS.md	Research and code constraints for AI collaborators
docs/	Current paper plan, novelty audit and dataset runbooks
results/	Immutable measured-result records and interpretation rules
literature_notes/	Paper index and persistent summaries
reports/	Generated dataset audits; not model results

Root Python and shell files remain intentionally stable because remote L20 commands invoke them directly. They are categorized in PROJECT_CONTEXT.md; moving them into src/, scripts/ or tests/ requires compatibility wrappers and regression testing.

Current paper question and experiment plan (2026-08-30)

The full staged plan is in the Omni speaker-floor experiment plan. It separates four questions that must not be conflated:

1. Does a frozen global Omni state encode aggregate wearer/others activity?
2. Can Omni features identify which visible person is speaking, rather than merely detect that somebody is speaking?
3. Does anonymous person-wise causal fusion reduce third-party false entries at matched wearer onset recall?
4. Does any suppression preserve addressed-to-wearer response recall/latency and wearer yield behavior, rather than making the model globally passive?

The immediate order is: run and audit the implemented competing-risk v2 event protocol, then freeze it; run speaker-grounding probes and wrong-person/time-shift controls; re-evaluate G, G+A, and G+A+pooled-V; then train an anonymous person-wise causal side branch. A person-memory adapter or Thinker LoRA is a second-stage experiment only after the simpler branch passes those diagnostic gates.

The literature boundary and novelty threats are recorded in the multiparty egocentric floor gap audit. The precise target is the intersection of egocentric wearer action, strict causality, speaker/addressee grounding, both entry and yield, and event-level evaluation. This repository does not claim the first multiparty AV turn-taking system, the first online multiparty EOT detector, or that a frozen Omni backbone is task-level end-to-end training.

The persistent paper index and Chinese summaries are maintained in literature_notes.

The pooled local-feature B1 experiment (frozen Qwen face crops + recent audio, native 2048D sidecars, optional trainable projection, four-epoch ablations) is documented in docs/local_feature_baseline.md.

For EasyCom encoder-only A / V / AV with no Thinker states, start with the EasyCom comparison commands. They reuse the verified local sidecars, train on Session 4–12, test on 3/2/1, and keep four epochs with no dev set.

The user-reported EasyCom encoder-only seed-0 results and complete supplied horizon curves are archived in the 2026-08-28 result record. It keeps causal-valid summaries separate from full-timeline curves and records the head-capacity/context caveats for comparisons with G+A and G+A+V.

The earlier engineering plan is speaker-aware Thinker and wearer-transition evaluation (2026-08-28): person-organized video chunks, frozen-Thinker local-evidence interventions, and onset/offset plus third-party-conversation diagnostics. These are planned experiments, not implemented or validated results.

The trainable speaker-fusion survey extends this to person-wise temporal branches, local-memory adapters and upper-layer Thinker LoRA on 8×L20, with matched-onset-recall false-entry evaluation and addressee-specific safeguards. It also records related-work and input-permission limits; none of these new variants has been implemented or measured yet.

The Stage-0 representation experiment is deliberately simple:

causal audio + video up to time t
                |
                v
     Qwen2.5-Omni-3B Thinker
                |
        fused state q_t [2048]
                |
                v
       trainable prediction head
                |
                v
  future logits [10 horizons, 3 classes]

There is no RNN after Qwen in this Stage-0 baseline. The Omni Thinker is its temporal model, and its only supervised component is a small 10 x 3 classification head. Later person-wise GRU/TCN branches and local-memory adapters are controlled methods for testing speaker grounding; they do not change what the Stage-0 result measures.

Current status

File	Status	Purpose
smoke_test.py	Verified on one NVIDIA L20	Text and 12-second EasyCom audio-video inference
easycom_manifest.py	Verified on EasyCom	Discover one-minute clips, preserve session timecodes, and check 5 Hz targets
qwen_omni_features.py	Implemented, not executed here	Load only the 3B Thinker and extract fused states from one AV window
streaming_turn_model.py	Verified on EasyCom	Produce one causal Omni state every 200 ms, with causal-prefix and open-stream modes
TurnPredictionHead	Trained and evaluated	Map [2048] to [10, 3] logits
train_turn_head.py	Verified on EasyCom	Train the frozen-Omni anticipation head with the official EasyCom split
infer_turn_head.py	Verified on EasyCom	Save EgoSpeak-format predictions and reproduce per-horizon AP/mAP
extract_turn_local_features.py	Verified on EasyCom; Ego4D extraction in progress	Extract person-crop and recent-audio Qwen sidecars
evaluate_floor_state_changes.py	Implemented; attribution revision required	Diagnose wearer entry/yield events from saved predictions
eval_omni_turn_lora.py	Implemented, remote validation pending	Evaluate Thinker LoRA from dense AV caches with the same AP/mAP protocol

The full Thinker input path, embedded audio, direct video decoding, FlashAttention-2, 5 Hz streaming extraction, cache replay, target alignment, head training, and EgoSpeak-compatible evaluation have now been exercised on EasyCom. The first Ego4D 8-second versus 20-second context ablation has also been recorded. Multi-seed evaluation remains in progress.

Current EasyCom results

The current single-run comparison was recorded on 2026-08-19. Both modes use the frozen Qwen2.5-Omni-3B Thinker, 2 FPS video, 5 Hz predictions, a 20-second causal context budget, the same 10 x 3 head architecture, and EgoSpeak's official EasyCom train/test split.

Representation	EgoSpeak full-timeline mean mAP	Strict causal mean mAP
Open-stream replay	58.316	58.326
Causal-prefix	58.830	58.841

Causal-prefix is 0.514 mAP points higher in the exact Table 2 protocol, but the difference is concentrated at 0.2--0.6 seconds. Open-stream is slightly higher at 1.6--2.0 seconds. These are not yet final paper values because the head-training variance has not been measured across multiple seeds.

The full per-horizon and per-class snapshot, machine-readable CSV, provenance, and follow-up checks are in results/easycom_open_stream_vs_causal_prefix_2026-08-19/.

EasyCom 8-second versus 20-second context

An additional single-run open-stream comparison recorded on 2026-08-23 gives 58.51937 full-timeline mean mAP with an 8-second context budget versus 58.316 with 20 seconds, a difference of +0.20337 points. The aggregate difference is small and hides a clear horizon tradeoff: 8 seconds is stronger at 0.2--0.8 seconds, whereas 20 seconds is stronger from 1.0 through 2.0 seconds. These runs must be repeated with identical additional seeds before choosing a context budget for the paper.

The raw 8-second console output, machine-readable comparison, and detailed interpretation are archived in results/easycom_open_stream_8s_vs_20s_2026-08-23/.

Last-4 query-layer learned mixture

A frozen Qwen2.5-Omni-3B experiment now reads the query states from the final four Thinker layers and learns four global softmax mixture weights before the same 10 x 3 anticipation head. With an 8-second open-stream context, 2 FPS video, 5 Hz outputs, and 25 head-training epochs, it reaches 73.45085 mAP at 0.2 seconds and 60.53423 mean mAP on official test Sessions 3, 2, and 1. Relative to the recorded single-final-layer 8-second run, this is +1.45317 at 0.2 seconds and +2.01486 mean mAP, with positive gains at every horizon.

The final oldest-to-newest layer weights are [0.246384, 0.248327, 0.241774, 0.263515]. Because they remain almost uniform, the result currently demonstrates the value of multi-layer query readout, not yet the value of learning the mixture. A fixed uniform last-4 mean is the next required control. The complete table, machine-readable metrics, provenance, and naming caveat are archived in results/easycom_frozen_3b_8s_last4_learned_mix_2026-08-26/.

Higher-capacity last-4 concat-MLP ablation

train_turn_head.py also supports a higher-capacity fusion control. It flattens each [4, 2048] last-layer stack to 8192 dimensions, applies LayerNorm -> Linear(8192, 2048) -> GELU, and then uses the unchanged anticipation MLP. This adds about 16.8 million fusion parameters, compared with four logits in the scalar-mixture experiment. Keep the seed and training recipe fixed for a clean capacity ablation:

CUDA_VISIBLE_DEVICES=0 \
OMP_NUM_THREADS=4 \
MKL_NUM_THREADS=4 \
OPENBLAS_NUM_THREADS=4 \
python train_turn_head.py \
  --feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack \
  --target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
  --output checkpoints/easycom_frozen_3b_8s_last4_concat_mlp_25ep.pth \
  --sessions 4 5 6 7 8 9 10 11 12 \
  --epochs 25 \
  --batch-size 512 \
  --learning-rate 7e-5 \
  --weight-decay 5e-5 \
  --warmup-epochs 10 \
  --warmup-factor 0.3 \
  --dropout 0.2 \
  --loss lstr-mce \
  --layer-fusion concat-mlp \
  --expected-mode open-stream-replay \
  --seed 0 \
  --device cuda:0

Evaluate with the same dense stacked test features:

CUDA_VISIBLE_DEVICES=0 python infer_turn_head.py \
  --feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack \
  --target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
  --checkpoint checkpoints/easycom_frozen_3b_8s_last4_concat_mlp_25ep.pth \
  --output-dir results/easycom_frozen_3b_8s_last4_concat_mlp_25ep \
  --expected-mode open-stream-replay \
  --batch-size 1024 \
  --device cuda:0

Treat this as a capacity ablation. If it improves, compare it against a parameter-matched single-layer MLP and use stronger regularization or a clean development session before attributing the gain to cross-layer fusion.

Four-epoch loss-structure ablation

Using the same frozen 8-second last-4 stack and concat-MLP readout, a three-seed fixed-budget comparison tested LSTR-MCE, independent sigmoid/BCE, and four-state softmax. All nine training sessions were used for exactly four epochs with no development set. On the causal-valid test metric, no objective is a decisive winner: BCE has the highest mean mAP (60.91733), while four-state CE has the highest 0.2-second wearer AP (78.72040). Relative to LSTR-MCE, both alternatives slightly improve the two speech classes while reducing background AP. The full protocol, per-seed summary, seed-0 horizon curves, and machine-readable data are archived in results/easycom_loss_structure_4ep_2026-08-26/.

R1/R2 transition-aware head ablation

The transition target for current state t and future horizon h is one exactly when the complete three-label activity row changes between t+h-1 and t+h. This includes onset, offset, speaker handover, and entering or leaving overlap. It never uses media or labels after the prediction target t+h.

Both variants share the activity MLP trunk and add a small binary 10-horizon transition readout:

• R1 supervised uses L = L_activity + 0.2 L_transition, where L_transition is BCEWithLogits with positive weight 3.
• R2 expected reward uses L = L_activity + 0.2 L_transition - 0.1 E[R_transition]. For transition probability p and target z, correctness reward is z p + (1-z)(1-p). Its expectation is computed exactly, weighted with the same positive weight 3, and normalized by total weight. There is no sampled policy gradient, so this is the low-variance control rather than a claim of interactive reinforcement learning.

Run the fixed four-epoch, no-development-set comparison with the same schedule as the loss-structure ablation:

FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame

for seed in 0 1 2
do
  CUDA_VISIBLE_DEVICES=0 \
  OMP_NUM_THREADS=4 \
  MKL_NUM_THREADS=4 \
  OPENBLAS_NUM_THREADS=4 \
  python train_turn_head.py \
    --feature-root "$FEATURE_ROOT" \
    --target-root "$TARGET_ROOT" \
    --output "checkpoints/easycom-transition-r1-4ep-seed-$seed.pth" \
    --sessions 4 5 6 7 8 9 10 11 12 \
    --epochs 4 \
    --batch-size 512 \
    --learning-rate 7e-5 \
    --weight-decay 5e-5 \
    --warmup-epochs 1.6 \
    --warmup-factor 0.3 \
    --dropout 0.2 \
    --loss lstr-mce \
    --transition-objective r1-supervised \
    --transition-loss-weight 0.2 \
    --transition-pos-weight 3 \
    --layer-fusion concat-mlp \
    --expected-mode open-stream-replay \
    --seed "$seed" \
    --device cuda:0

  CUDA_VISIBLE_DEVICES=0 \
  OMP_NUM_THREADS=4 \
  MKL_NUM_THREADS=4 \
  OPENBLAS_NUM_THREADS=4 \
  python train_turn_head.py \
    --feature-root "$FEATURE_ROOT" \
    --target-root "$TARGET_ROOT" \
    --output "checkpoints/easycom-transition-r2-4ep-seed-$seed.pth" \
    --sessions 4 5 6 7 8 9 10 11 12 \
    --epochs 4 \
    --batch-size 512 \
    --learning-rate 7e-5 \
    --weight-decay 5e-5 \
    --warmup-epochs 1.6 \
    --warmup-factor 0.3 \
    --dropout 0.2 \
    --loss lstr-mce \
    --transition-objective r2-expected-reward \
    --transition-loss-weight 0.2 \
    --transition-pos-weight 3 \
    --transition-reward-weight 0.1 \
    --layer-fusion concat-mlp \
    --expected-mode open-stream-replay \
    --seed "$seed" \
    --device cuda:0
done

Evaluate all six checkpoints. In addition to the unchanged EgoSpeak activity metrics, inference writes transition_pred arrays and prints transition AP, precision, recall, and F1 at threshold 0.5:

for variant in r1 r2
do
  for seed in 0 1 2
  do
    CUDA_VISIBLE_DEVICES=0 python infer_turn_head.py \
      --feature-root "$FEATURE_ROOT" \
      --target-root "$TARGET_ROOT" \
      --checkpoint "checkpoints/easycom-transition-$variant-4ep-seed-$seed.pth" \
      --output-dir "results/easycom-transition-$variant-4ep-seed-$seed" \
      --expected-mode open-stream-replay \
      --batch-size 1024 \
      --device cuda:0
  done
done

Compare R1 and R2 against the existing three-seed LSTR-MCE baseline under the same four-epoch budget. The primary paper outcome remains activity mAP; the transition metrics diagnose whether an activity improvement is genuinely associated with better boundary prediction.

The preliminary seed-0/1 results are archived in results/easycom_transition_auxiliary_4ep_2026-08-27/. Neither R1 nor R2 establishes a meaningful activity-mAP gain. At 0.2 seconds, transition prevalence is 11.21718%, while transition AP is approximately 15.3%, indicating weak but non-random ranking. R2 improves activity mean mAP over R1 by only 0.01162 points and transition AP by 0.16660 points. Seed 2 is still required before freezing this as a complete null ablation.

Ego4D local-feature ablation family

To test whether the recent-audio / per-person-visual gains transfer from EasyCom, see the Ego4D local-feature experiment guide. It reuses existing global Omni features and the EgoSpeak manifest, extracts local A/V directly from raw clips, and runs G / G+A / G+V / G+A+V for four epochs, without a dev set. run_ego4d_local_baselines.py preflights all train/test files and encoder contracts before training; the summary reports matched-seed deltas, including AV minus audio. No new accuracy claim is implied by adding the code.

The global train/<uid>.pt and test/<uid>.pt layout is now read directly by extraction, training and inference; no symlink index is needed. extract_ego4d_local_multigpu.py distributes full clips across eight independent GPU workers, with per-GPU logs, validated resume, and foreground supervision. See the guide's L20 commands for the /data1/yfzx/wangx paths.

For encoder-only A / V / AV, without Thinker states, see the Ego4D guide or the EasyCom-first guide. prepare_encoder_only_features.py packs complete local vectors and validity bits on CPU (pass --dataset easycom for EasyCom; default remains Ego4D). run_ego4d_encoder_only.py / run_easycom_encoder_only.py train/evaluate four-epoch heads with no dev set. This native-concatenation baseline is not capacity- or context-matched to the Thinker baseline.

The first epoch-budget sweep shows severe capacity overfitting. Four epochs reach 73.42020 at 0.2 seconds and a new frozen-Omni best of 60.87144 mean mAP, whereas 12 and 50 epochs fall to 57.98809 and 53.03515 despite lower training losses. The 4-epoch concat-MLP is effectively tied with the scalar mixture at 0.2 seconds (-0.03065) but improves mean mAP by +0.33721, mainly at 1.2--1.6 seconds. Because the runs used different total-length cosine schedules and the official test sessions were inspected, this is a diagnostic capacity result rather than a valid final epoch selection. Full tables and protocol caveats are archived in results/easycom_frozen_3b_8s_last4_concat_mlp_epoch_budget_2026-08-26/.

Frozen ROMA last-four-layer transfer

Three frozen ROMA/Qwen2.5-Omni-7B readout runs were recorded on 2026-08-27 using the same 8-second open-stream, 2 FPS, 5 Hz EasyCom evaluation path. A four-epoch fixed mean over the last four [3584] query states is the best ROMA control at 69.21863 mAP for 0.2 seconds and 58.90402 mean mAP. A 51.4M-parameter concat-MLP reaches only 68.65486 / 56.52425 after four epochs; relative to the design- and epoch-matched frozen Qwen-3B concat model it is -4.76534 at 0.2 seconds and -4.34719 in mean mAP. This is not parameter-matched because the 3584-dimensional ROMA fusion is much wider.

This is a negative-transfer diagnostic, not evidence that ROMA is universally worse: the project used Qwen-style open-stream replay rather than ROMA's native one-second streaming training recipe. The practical decision is nevertheless clear: retain frozen Qwen2.5-Omni-3B as the main backbone and stop tuning the wide ROMA concat head. Full horizon tables, class AP, validity caveats, JSON, and a formatted workbook are archived in results/easycom_roma_frozen_last4_2026-08-27/.

The related-work synthesis and proposed next method—recent-200ms payload readout plus structured future-floor projection—are documented in reports/roma_negative_transfer_and_future_floor_projection_2026-08-27.md.

Current Ego4D context-length results

The following single-seed snapshot was recorded on 2026-08-22. Both variants use open-stream replay, 2 FPS Decord video, 5 Hz predictions, the same frozen Qwen2.5-Omni-3B Thinker and independently trained 10 x 3 heads. They differ only in the rolling causal-memory budget: 8 seconds versus 20 seconds. Both runs use the same EgoSpeak Ego4D train/test manifest and test timeline.

Context	EgoSpeak full-timeline mean mAP	Strict causal mean mAP
8 seconds	65.5943	65.6920
20 seconds	63.1408	63.2298
20s minus 8s	-2.4535	-2.4622

The strict-causal per-horizon comparison is:

Horizon	8s mAP	20s mAP	20s minus 8s
0.2s	74.2213	71.7403	-2.4810
0.4s	71.0180	68.5945	-2.4234
0.6s	68.3746	65.9287	-2.4459
0.8s	66.3190	63.8606	-2.4584
1.0s	64.7470	62.3329	-2.4140
1.2s	63.6416	61.2290	-2.4126
1.4s	62.8697	60.4035	-2.4662
1.6s	62.3142	59.7870	-2.5272
1.8s	61.8705	59.3678	-2.5027
2.0s	61.5444	59.0541	-2.4903
Mean	65.6920	63.2298	-2.4622

For the exact full-timeline protocol, averaging class AP over the ten horizons gives:

Class	8s mean AP	20s mean AP	20s minus 8s
Background	69.5063	67.2409	-2.2653
Camera wearer speaking	59.4005	56.5784	-2.8222
Other people speaking	67.8760	65.6030	-2.2730

The 8-second representation is better at every horizon and for every class. The approximately 2.4--2.5 point gap is nearly constant across anticipation horizons, while the largest class degradation is for camera-wearer speech. This pattern does not support a long-horizon benefit from the additional 12 seconds of context. A plausible hypothesis is that excessive history dilutes locally predictive turn-taking cues in the final frozen Thinker state, but the single-seed comparison cannot establish that mechanism. Both context settings must be repeated with the same additional seeds before treating the difference as a final paper result.

Why Omni-only instead of Omni + RNN

EgoSpeak separately extracts visual and audio features and then uses GRU, LSTR, or Mamba for temporal modeling:

ResNet50 video feature + wav2vec2 audio feature -> temporal model -> prediction

Qwen2.5-Omni already contains:

• audio and visual encoders;
• joint audio-video temporal positions;
• a causal decoder-only Thinker with KV cache;
• pretrained acoustic, visual, and semantic knowledge.

Adding a GRU after a continuously processed Omni sequence would model history twice and would make it harder to attribute an improvement to Omni. The primary comparison is therefore:

EgoSpeak: ResNet50 + wav2vec2 + GRU
Ours:     Qwen2.5-Omni Thinker + small prediction head

An Omni-feature-plus-GRU model can be kept as an ablation, but it is not the main architecture.

EgoSpeak contract that this project reproduces

The inspected reference repository is /Volumes/760P/Code/EgoSpeak. It contains no Qwen or Omni implementation and no raw-media feature extraction scripts. It distributes precomputed arrays instead:

DATASET/
|- rgb_kinetics_resnet50/   # [T, 2048]
|- audio_wav2vec2/          # [T, 5120]
`- target_perframe/         # [T, 3]

The useful part for this project is the exact task interface.

Time and classes

Both EasyCom and the EgoSpeak EGO4DSHUFFLE split use 5 FPS, so one task timestep is 0.2 seconds. The three output columns are:

0: background
1: camera wearer speaking
2: other people speaking

They are not strictly one-hot. Background is exclusive, but simultaneous camera-wearer and other-person speech is stored as [0, 1, 1]. The inspected EasyCom features contain 7,711 such overlap steps. Preserve the full multi-positive target rather than applying argmax.

Relevant EgoSpeak files:

• LSTR/data/data_info.json: FPS, classes, and session splits;
• MiniROAD/configs/EasyCom/miniroad_aa_easycom.yaml: 128-step training windows, stride 4, three classes, and anticipation length 10;
• MiniROAD/datasets/dataset.py: feature loading and future-label indexing;
• MiniROAD/criterions/loss.py: anticipation loss;
• MiniROAD/trainer/eval.py: per-horizon AP and mean anticipation mAP.

Future target alignment

For current task index i, the prediction target is:

target[i + 1 : i + 11]

Thus the ten output rows represent:

t+0.2, t+0.4, t+0.6, ..., t+2.0 seconds

If states[i] is verified to correspond to EgoSpeak task index i, the offline head-training tensors should be constructed as follows:

target = target_perframe                            # [T, 3], multi-positive
usable = min(len(states), len(target)) - 10

x = states[:usable]                                 # [N, 2048]
y = stack(
    [target[h : h + usable] for h in range(1, 11)],
    axis=1,
)                                                   # [N, 10, 3]

Do not silently assume that the first saved Omni state matches label index zero. EgoSpeak does not publish the script that generated its original ResNet/wav2vec2 features, so the exact zero-time convention is not documented. Before training, inspect a short clip with known speech boundaries and test the candidate mappings states[i] -> target[i] and states[i] -> target[i+1]. Store the selected offset in every feature file/checkpoint.

Splits

Use the published split, not a random frame split.

EasyCom:

• train, preserving EgoSpeak's list: Session_7, Session_11, Session_10, Session_6, Session_9, Session_4, Session_8, Session_5, Session_12;
• test, preserving EgoSpeak's list: Session_3, Session_2, Session_1.

This is the EgoSpeak comparison split. The EasyCom dataset repository itself recommends a different train/validation/test split; do not substitute that split when reproducing EgoSpeak Table 2.

Ego4D Audio-Visual Diarization:

• use the EGO4DSHUFFLE key;
• 346 training clips;
• 87 test clips;
• each clip is approximately five minutes.

Never mix timesteps from the same session across training and validation.

EasyCom raw-media layout and manifest

The extracted release used here has this relevant structure:

extracted_easycom/
`- Main/
   |- Video_Compressed/
   |  |- Session_1/
   |  |  |- 00-00-000.mp4
   |  |  |- 01-00-000.mp4
   |  |  `- ...
   |  `- Session_2 ... Session_12/
   |- Glasses_Microphone_Array_Audio/
   |- Close_Microphone_Audio/
   |- Speech_Transcriptions/
   `- Metadata/

Video_Compressed is the AR-glasses wearer's egocentric stream with embedded stereo AAC audio. Each session is divided into approximately one-minute clips; the mm-ss-SSS filename is the clip's start time on the original recording timeline. Some intervals are absent because EasyCom redacted one-minute segments. These raw time gaps are expected.

EgoSpeak does not preserve those empty intervals. Its appendix states that it crops every EasyCom clip to retain only the 1920-pixel-wide video region and then merges the retained clips into one dense video per session. Therefore the EgoSpeak-compatible label time for a clip is its cumulative position in the concatenation, not the mm-ss-SSS source time. The manifest stores both.

Build a raw-video manifest and, when the EgoSpeak labels are available, check their duration at 5 Hz:

python easycom_manifest.py \
  --dataset-root /data1/yfzx/wangx/Datasets/easycom/extracted_easycom \
  --target-root /path/to/EasyCom/target_perframe \
  --output manifests/easycom.json

Omit --target-root for the first media-only inventory. The manifest reports raw-source discontinuities as informational and computes each clip's dense EgoSpeak concatenation offset and corresponding 5 Hz target index. For the audio-only ablation, use the same embedded glasses audio as A+V (extract it to mono WAV if needed); do not switch to close microphones, because that changes the sensor and makes the modality ablation unfair.

Audit downloaded EgoSpeak arrays, and optionally compare every session length against the raw manifest:

python audit_egospeak_features.py \
  --feature-root /path/to/EgoSpeak/dataset/EasyCom \
  --raw-manifest manifests/easycom.json \
  --output reports/egospeak_easycom_audit.json

The inspected release contains 95,453 synchronized steps across all 12 sessions: 5.302944 hours at 5 Hz. Every session has identical RGB, flow, audio, and target length. Shapes are [T,2048], [T,2048], [T,5120], and [T,3].

Recommended comparison matrix

Experiment	Visual sampling	Audio	Output rate
EgoSpeak official A+V	5 Hz ResNet features	5 Hz wav2vec2 features	5 Hz
Omni audio-only	none	raw glasses-wearer waveform	5 Hz
Omni A+V primary	2 FPS	raw embedded waveform	5 Hz
Omni A+V FPS ablation	5 FPS	raw embedded waveform	5 Hz

Two FPS is the primary Omni setting because it matches Qwen's default video sampling and permits longer context. Five FPS is the visual-rate ablation. Input feature rates need not match EgoSpeak; the split, causal prediction times, three classes, ten target horizons, and evaluator must match exactly.

Prepare an EgoSpeak-compatible EasyCom session

Do not pass the raw one-minute files independently to Omni and do not preserve the gaps implied by their filenames. EgoSpeak concatenates the retained clips into a dense session timeline and crops the participant-ID banner on the left.

prepare_easycom_session.py reproduces this policy and creates videos whose stored frames already lie on a fixed session-global 2 FPS or 5 FPS grid. This prevents Qwen's per-window video loader from choosing a new adaptive set of frames for every overlapping prefix.

Session 1 has been checked against the local release:

24 retained clips
27,768 source frames at 20 FPS
/ 4 frames per task step
= 6,942 steps at 5 Hz
= target_perframe/Session_1.npy length exactly

Preview the commands and validate frames against the target without encoding:

python prepare_easycom_session.py \
  --session-dir /Volumes/MX500B/Code/easycom/Main/Video_Compressed/Session_1 \
  --output-dir prepared_easycom/Session_1 \
  --target /Volumes/MX500B/Code/EgoSpeak/dataset/EasyCom/target_perframe/Session_1.npy \
  --fps 2 5 \
  --dry-run

Remove --dry-run to run FFmpeg. The outputs are:

Session_1_2fps.mp4     primary Omni A+V input
Session_1_5fps.mp4     visual-rate ablation
Session_1.json         clip order, exact T, crop and sampling provenance
Session_1.ffconcat     reproducible dense concatenation order

The target length must equal the number of complete 5 Hz intervals, floor(sum(source_frames) / 4). If a session ends with one to three residual 20 Hz frames, the target-derived duration discards only that incomplete final interval; the discarded frame count is recorded in the sidecar.

Each clip's decoded video and audio are trimmed to the exact duration implied by its video frame count before concatenation. This prevents AAC priming/tail padding from accumulating at the 24 clip boundaries and causing progressive audio/video drift. The session-global visual grid is padded defensively at EOF and then trimmed by frame index, so it contains exactly ceil(target_steps / 5 * video_fps) frames on every FFmpeg version.

On the remote L20 machine, use its own dataset paths. For example:

python prepare_easycom_session.py \
  --session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_1 \
  --output-dir /data1/yfzx/wangx/Datasets/easycom/omni_prepared/Session_1 \
  --target /path/to/EgoSpeak/dataset/EasyCom/target_perframe/Session_1.npy \
  --fps 2 5

This is the command that performs the potentially long FFmpeg processing; it has not been run by the code authoring workflow.

If encoding completed but a later validation/metadata step failed, update the script and reuse the existing MP4 without encoding it again:

python prepare_easycom_session.py ... --fps 2 --reuse-existing

What Qwen's native 2-second streaming design means

Qwen2.5-Omni was designed for block-wise multimodal prefill. Its technical report states that the audio and visual encoders are modified for block-wise processing and that audio self-attention is performed in independent two-second blocks. The released 3B configuration uses seconds_per_chunk=2, position_id_per_seconds=25, and audio n_window=100.

Conceptually, a native online input is organized as follows:

open one user/media turn
  encode/append AV [0,2]       -> commit Thinker KV
  encode/append AV [2,4]       -> commit Thinker KV
  encode/append AV [4,6]       -> commit Thinker KV
  ...
close the media turn           -> generate text and/or speech

Within each temporal group, video tokens precede the aligned audio tokens. TMRoPE assigns 25 temporal positions per second, or 50 positions per native two-second group, so audio and video share a physical-time coordinate system. Completed blocks are appended to one causal Thinker sequence; the long-range history is modeled by the Thinker rather than by cross-block full attention in the modality encoders.

This architecture is block-causal, not frame-causal. Attention inside an audio block is non-causal. It is therefore invalid to encode the complete [0,2] block and claim that its representation at 0.2 seconds saw no future:

encode complete [0,2] AV -> read state at 0.2 s  # future leakage

For this project's 5 Hz targets, the current incomplete block must instead be encoded only through the prediction time. These 0.2-second partial-block readouts are an extension made for EgoSpeak; the Qwen paper does not define a canonical hidden state every 200 ms.

The public Transformers and local Qwen web-demo APIs normally receive a completed media object and call the processor/model once. They retain the model's block-wise masks and token organization, but do not expose the full persistent live-session implementation used by the hosted real-time product.

Native Qwen streaming versus causal-prefix

The two extraction modes are both causal with respect to raw observations, but they define different model inputs. causal-prefix is a sliding-window reference invented for this experiment; it is not Qwen's native streaming execution model.

Property	causal-prefix	Native Qwen streaming idea
Session	New inference at every 0.2 s	One open media turn
Media organization	One complete [max(0,t-M),t] item	Fixed source-aligned 2 s blocks
Modality encoding	Re-encode the complete window	Encode/commit each completed block once
Thinker KV	Start empty for every state	Append to persistent KV
TMRoPE	Recomputed from each window origin	Advances with the stream
History	Exact rolling M seconds	Accumulates until a serving/context policy intervenes
5 Hz readout	Close/read a new turn each time	Not specified by the original model

The difference becomes especially important after a sliding window fills. At t=20.8 with M=20, causal-prefix constructs one new media item covering:

[0.8,20.8]
  -> encoder blocks [0.8,2.8], [2.8,4.8], ..., [18.8,20.8]

At t=21.0, every encoder boundary moves again:

[1.0,21.0]
  -> encoder blocks [1.0,3.0], [3.0,5.0], ..., [19.0,21.0]

Native streaming keeps the source-aligned partition instead:

[0,2], [2,4], ..., [18,20], current partial [20,20.8]

Because Qwen's audio encoder uses block-local non-causal attention, moving a block boundary changes the representation even when most raw samples are the same. Window-relative TMRoPE, convolution/padding boundaries, chat closure, and BF16/attention execution add further differences. This is why exact KV replay can be numerically identical to its own from-scratch replay while still differing from causal-prefix; the modes are evaluating different token and encoder histories.

Use the modes according to the scientific question:

• use open-stream-replay for the primary online experiment, training, and deployment-aligned evaluation;
• use causal-prefix as an expensive continuous-window reference and short-clip ablation;
• never train a head on one representation and deploy it on the other without retraining, because their feature distributions are not interchangeable.

Causal 5 Hz extraction modes

streaming_turn_model.py implements two modes.

1. causal-prefix: continuous-window reference

At every 200 ms, independently process the causal context ending at the current time:

[max(0, t-20 s), t] AV -> Omni -> q_t

Properties:

• strict no-future input;
• one continuous media window per prediction;
• no cross-forward cache;
• very expensive because overlapping context is recomputed;
• useful for checking continuous sliding-window semantics and causality;
• not the native Qwen block-streaming execution model.

Example:

python streaming_turn_model.py \
  --video /path/to/easycom_session.mp4 \
  --use-audio-in-video \
  --mode causal-prefix \
  --start 0 \
  --end 20 \
  --memory-seconds 20 \
  --output features/easycom_20s_prefix.pt

2. open-stream-replay: recommended efficient mode

This mode keeps one user/media turn open across the whole bounded-memory segment. The first block contributes the chat and AV opening markers once; later completed two-second blocks contribute only interleaved audio/video payload tokens:

open AV prefix
  completed [0,2] payload       -> commit KV
  current   [2,2.2] payload     -> temporary branch -> close/read q_t
  current   [2,2.4] payload     -> temporary branch -> close/read q_t
  ...
  completed [2,4] payload       -> commit KV
AV/chat closure                 -> never committed; readout branch only

This matches the core Qwen design choices: fixed two-second AV groups, one open media turn, completed-block commits, and incremental Thinker KV. It makes two practical changes for this dataset:

1. the current 0--2 second partial block is evaluated speculatively every 0.2 seconds and discarded unless it reaches a complete block boundary;
2. the KV history is bounded and rebuilt from recent complete blocks rather than accumulating for an entire multi-minute session. Complete-block AV embeddings are retained in a bounded ring, so rolling rebuilds rerun only the Thinker and do not rerun the processor, audio tower, or video tower.

It should therefore be described as bounded Qwen-style open streaming, not as bit-for-bit reproduction of an unbounded official serving session.

The audio encoder accepts a shorter final chunk with padding and an actual-length mask. For the two-frame visual patch requirement, strict-grid mode duplicates only the most recent past frame when necessary.

Run a four-second validation before using it for extraction:

python streaming_turn_model.py \
  --model-id /path/to/Qwen2.5-Omni-3B \
  --video prepared_easycom/Session_1/Session_1_2fps.mp4 \
  --use-audio-in-video \
  --mode open-stream-replay \
  --start 0 --end 4 \
  --video-fps 2 \
  --strict-fixed-video-grid \
  --verify-every 1 \
  --verify-continuous-every 1 \
  --flash-attention-2 \
  --output features/Session_1_open_first_4s.pt

cache_replay must be identical to replaying the same open stream from an empty cache. continuous_window_semantics compares it with one normal continuous Qwen AV input. Do not assume equivalence: the modality encoders are still invoked per tail block in the efficient path, so this numerical check is required even though artificial user-turn boundaries have been removed.

Whole-session prediction and the 20-second limit

Twenty seconds is a practical memory/context setting for the 3090, not the maximum video duration and not a limit specified by Qwen. The original append-only idea eventually reaches both linear KV-memory growth and the Thinker's 32,768-token context limit, so an unbounded multi-minute stream is not practical. The complete session is processed sequentially and a prediction is emitted every 0.2 seconds.

open-stream-replay uses blockwise rolling memory. With a 20-second budget it rebuilds the open cache from the latest nine completed blocks (18 seconds) when a new block starts, then adds the current 0--2 second tail. Its effective history therefore varies from 18 to 20 seconds and never drops to an empty 0.2-second context at a 20-second boundary. The nine blocks' prepared AV embeddings and local TMRoPE coordinates are cached once; strict window replay still recomputes the Thinker KV so retained blocks no longer carry indirect influence from observations that fell out of the window.

For a short native-stream fidelity check, set --memory-seconds to at least the tested clip duration. For example, the options --start 0 --end 40 --memory-seconds 40 prevent a rolling rebuild during that 40-second clip. Do not extend this policy to complete EasyCom/Ego4D sessions: KV memory grows linearly, and this implementation clones the cache for speculative partial blocks and readout branches.

For the paper experiment, drive the output length from the preparation sidecar rather than a floating MP4 duration:

python streaming_turn_model.py \
  --session-metadata prepared_easycom/Session_1/Session_1.json \
  --use-audio-in-video \
  --mode open-stream-replay \
  --memory-seconds 20 \
  --video-fps 2 \
  --strict-fixed-video-grid \
  --verify-every 25 \
  --verify-continuous-every 25 \
  --output features/Session_1_omni_2fps.pt

The sidecar selects the matching fixed-FPS MP4 and sets N to its exact num_steps. Alternatively pass --video ... --target ... explicitly. A raw 20 FPS MP4 with --video-fps 2 is rejected when --strict-fixed-video-grid is enabled.

In strict A+V mode the extractor does not delegate temporal frame selection to the generic video utility. It reads the already-fixed MP4 grid explicitly and assigns both video frames and audio samples with half-open intervals [start, end). Thus a frame exactly on a two-second boundary belongs only to the following block. It duplicates the most recent available frame when the visual tower needs an even frame count. Audio is loaded once at 16 kHz and sliced at the same causal boundary. The processor still receives use_audio_in_video=True, so Qwen's audio/video token interleaving is retained. Long runs use tqdm and update the progress bar after every completed state, showing percentage, media time, elapsed time, ETA, and throughput. Pass --no-progress to disable it. The Thinker-only path also suppresses Qwen's irrelevant speech-output system prompt warning.

Before a full session, run only the first four seconds and inspect both check types:

python streaming_turn_model.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --video /data1/yfzx/wangx/Datasets/easycom/omni_prepared/Session_1/Session_1_2fps.mp4 \
  --use-audio-in-video \
  --mode open-stream-replay \
  --start 0 --end 4 \
  --video-fps 2 \
  --strict-fixed-video-grid \
  --verify-every 1 \
  --verify-continuous-every 1 \
  --flash-attention-2 \
  --output features/Session_1_first_4s_check.pt

Do not launch all sessions unless cache_replay cosine is nearly 1. For open-stream-replay, a materially lower continuous_window_semantics cosine means that the custom payload stitching is not equivalent to one continuous Qwen input; it must not be dismissed as a media-turn-boundary effect because this mode intentionally keeps a single media turn open.

Video can be sampled at 2 FPS while predictions remain at 5 Hz. Audio and the causal boundary update every 200 ms; consecutive prediction times may use the same most-recent visual frame. A frame with timestamp later than t must never be included.

Direct extraction without a prepared 2 FPS MP4

streaming_turn_model_directly.py removes the offline video-transcoding requirement. It reads original frame PTS with Decord, constructs a global 2 FPS lattice, maps every lattice point to the latest source frame whose PTS is not later than that point, and passes explicit frame lists to Qwen. Audio is decoded once to 16-kHz mono and sliced at exact causal boundaries. Therefore no FFmpeg fps filter or second H.264/AAC encode can alter the experiment.

Validate one Ego4D clip without loading Qwen:

python streaming_turn_model_directly.py \
  --video /data/wangx/Datasets/ego4d/v1/clips/CLIP_UID.mp4 \
  --target /data/wangx/Datasets/ego4d/ego4d_target_frame/CLIP_UID.npy \
  --video-fps 2 \
  --media-only

Run a short native persistent-KV check:

python streaming_turn_model_directly.py \
  --model-id /data/wangx/Models/Qwen2.5-Omni-3B \
  --video /data/wangx/Datasets/ego4d/v1/clips/CLIP_UID.mp4 \
  --end 40 \
  --mode open-stream-replay \
  --memory-seconds 0 \
  --video-fps 2 \
  --flash-attention-2 \
  --device cuda:0 \
  --output features/direct/CLIP_UID_first_40s.pt

Here --memory-seconds 0 means that completed-block Thinker KV persists for the entire requested interval. Do not use this setting blindly for complete multi-minute recordings: it has linear KV growth and remains subject to the Thinker's context limit. A full bounded-memory dataset run should explicitly use the experimental budget, for example --memory-seconds 20.

EasyCom can be read directly as the dense concatenation of retained clips; filename timecode gaps are ignored and the rightmost 1920x1080 region is kept:

python streaming_turn_model_directly.py \
  --model-id /path/to/Qwen2.5-Omni-3B \
  --easycom-session-dir /path/to/Main/Video_Compressed/Session_1 \
  --target target_frame/Session_1.npy \
  --mode open-stream-replay \
  --memory-seconds 20 \
  --video-fps 2 \
  --flash-attention-2 \
  --device cuda:0 \
  --output features/direct/Session_1.pt

The direct reader defaults to Decord. On NVIDIA systems, install PyNvVideoCodec and select the NVDEC backend explicitly:

python -m pip install PyNvVideoCodec

CUDA_VISIBLE_DEVICES=0 python streaming_turn_model_directly.py \
  --model-id /path/to/Qwen2.5-Omni-3B \
  --easycom-session-dir /path/to/Main/Video_Compressed/Session_1 \
  --end 24 \
  --mode open-stream-replay \
  --memory-seconds 20 \
  --video-fps 2 \
  --frame-cache-size 64 \
  --decoder pynvcodec \
  --decoder-gpu-id 0 \
  --flash-attention-2 \
  --device cuda:0 \
  --output features/direct/Session_1_24s_pynvcodec.pt

The NVDEC backend uses ffprobe for exact display-order PTS indexing, one reconfigurable SimpleDecoder for pixel decode, and the existing ffmpeg PCM audio path. --decoder-gpu-id is relative to CUDA_VISIBLE_DEVICES, so it is normally 0 even when the worker is assigned a nonzero host GPU. Decoder RGB conversion can differ slightly; validate a short Decord/PyNvVideoCodec pair before extracting every session, and do not mix decoder backends across train/test splits.

Persistent-model Ego4D batch extraction

extract_ego4d_batch.py loads Qwen and its processor once, then extracts the selected EgoSpeak manifest rows sequentially. Each clip receives a new media reader, Thinker KV cache, TMRoPE origin, and rolling-memory state; only the immutable model weights and processor are reused. Manifest indices are one-based data-row indices and do not count the TSV header.

For example, process manifest samples 81 through the end on visible GPU 0:

CUDA_VISIBLE_DEVICES=0 \
OMP_NUM_THREADS=4 \
MKL_NUM_THREADS=4 \
OPENBLAS_NUM_THREADS=4 \
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
python extract_ego4d_batch.py \
  --data-root /data1/yfzx/wangx/Datasets/ego4d \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --start-index 81 \
  --memory-seconds 20 \
  --video-fps 2 \
  --decoder decord \
  --dtype bfloat16 \
  --flash-attention-2 \
  --device cuda:0

Completed non-empty .pt files are skipped. Each new output is first written to a process-specific .partial path and atomically renamed after extraction, so rerunning the same range resumes safely.

For CPU Decord, the direct reader intentionally indexes each clip and releases that temporary reader, then keeps an LRU of at most two live readers during extraction. Each reader defaults to one decode thread. These limits avoid one worker retaining a thread pool for every one-minute EasyCom clip and can be overridden with --decoder-threads and --max-open-readers. The recommended full-run settings are:

--decoder decord --decoder-threads 1 --max-open-readers 2
--frame-cache-size 64

Matching states.shape[0] to the target is necessary but not sufficient. The direct reader also validates EasyCom's 20-Hz source inventory, source PTS monotonicity, dense duration, crop bounds, audio sample count, and causal frame-to-grid mapping. The original prepare scripts remain useful only as an optional decode-speed/reproducibility cache.

Saved streaming tensors

Each .pt file contains:

• times: [N], prediction times in seconds;
• states: [N, 2048], causal fused Thinker states;
• label_indices: [0, ..., N-1], direct target_perframe row alignment;
• valid_10_horizon_steps: N-10, because the last ten current rows do not have all future labels;
• mode: causal-prefix or open-stream-replay;
• stream_representation: explicit input-organization metadata;
• config: step, native block, memory, FPS, and pixel settings;
• verification: cached-versus-exact numerical checks;
• input_video_fps and video_sampling_grid: sampling provenance;
• media_interval_policy: fixed-grid AV uses half-open [start, end) ownership;
• logits: optional [N, 10, 3] output if a trained head is supplied;
• probabilities: optional softmax probabilities.

Training the 10 x 3 head

The head must be trained. It is not part of pretrained Qwen.

The first experiment should freeze all Qwen parameters and train only:

LayerNorm(2048)
-> Linear(2048, 1024)
-> GELU
-> Dropout
-> Linear(1024, 30)
-> reshape [10, 3]

easycom_head.py defines the head and loss; they remain re-exported by streaming_turn_model.py for compatibility:

from streaming_turn_model import (
    TurnPredictionHead,
    egospeak_multihorizon_loss,
)

The supplied LSTR config uses Adam, 50 epochs, learning rate 7e-5, weight decay 5e-5, dropout 0.2, and 10 warmup epochs followed by cosine decay. train_turn_head.py uses those defaults. Its batch size is 512 causal states, not LSTR's 16 temporal windows, because the two model inputs have different shapes.

Train once on the official nine sessions with a fixed 50-epoch protocol:

python train_turn_head.py \
  --feature-root features/open_2fps \
  --target-root target_frame \
  --output checkpoints/open_2fps_head.pth \
  --device cuda:0

The default loss is lstr-mce, exactly matching LSTR's MCE treatment of a multi-positive [0,1,1] row: it sums both active negative-log-softmax terms without normalizing the target. The previous L2-normalized objective belongs to MiniROAD and is available only as the explicit --loss miniroad-normalized ablation.

Two label-structure ablations are also available without changing the saved three-column target files:

• --loss independent-bce keeps a 10 x 3 head, trains the three labels with independent sigmoid/BCE, and evaluates the sigmoid marginals directly;
• --loss four-state-ce changes the head to 10 x 4 and maps each target to silence, wearer_only, others_only, or overlap. At evaluation time it returns the original three marginal scores as P(silence), P(wearer_only)+P(overlap), and P(others_only)+P(overlap).

The four-state marginals are intentionally not renormalized. Keep features, split, fusion, optimizer, schedule, and seeds fixed when comparing these losses. For checkpoint selection, reserve one official training session such as Session 12 with --dev-sessions 12; Sessions 1--3 remain test-only.

Recommended experimental procedure:

1. Freeze Qwen and extract all training-session states offline.
2. Verify state/label time offset and truncate only the final ten invalid current positions.
3. Keep EgoSpeak targets as float multi-positive tensors [N, 10, 3]; never convert them with argmax.
4. For comparison with the supplied LSTR shell, train with the default egospeak_multihorizon_loss/lstr-mce objective.
5. Select hyperparameters using only training sessions. For EasyCom, reserve a training session for development or use session-level cross-validation; never tune on Sessions 1--3.
6. After choosing the epoch/hyperparameters, optionally retrain on all nine EasyCom training sessions and evaluate the fixed model once on the test set.

Class-balanced/focal loss, a learned turn-query token, and LoRA are later ablations. They should not replace the frozen-Omni linear/MLP-head baseline.

Offline frozen states with transition-sparse supervision

This is the clean sampling-only control. It does not reconstruct cached AV tokens and does not run the Thinker during training. It loads the same dense 5 Hz frozen states used by the standard head baseline, then retains only the target_index values listed by the existing transition-sparse cache manifests. The representation, head architecture, optimizer, 50 epochs, loss, targets, and final dense test evaluator are otherwise unchanged.

Set FROZEN_FEATURE_ROOT to the exact feature directory used by the matching dense-supervision baseline. Do not compare sparse 8-second states with a dense 20-second baseline (or vice versa):

FROZEN_FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_20s_decord
TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
SPARSE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s

CUDA_VISIBLE_DEVICES=3 \
python train_turn_head.py \
  --feature-root "$FROZEN_FEATURE_ROOT" \
  --target-root "$TARGET_ROOT" \
  --sparse-anchor-cache-dir "$SPARSE_ROOT/Session_7" \
  --sparse-anchor-cache-dir "$SPARSE_ROOT/Session_11" \
  --sparse-anchor-cache-dir "$SPARSE_ROOT/Session_10" \
  --sparse-anchor-cache-dir "$SPARSE_ROOT/Session_6" \
  --sparse-anchor-cache-dir "$SPARSE_ROOT/Session_9" \
  --sparse-anchor-cache-dir "$SPARSE_ROOT/Session_4" \
  --sparse-anchor-cache-dir "$SPARSE_ROOT/Session_8" \
  --sparse-anchor-cache-dir "$SPARSE_ROOT/Session_5" \
  --sparse-anchor-cache-dir "$SPARSE_ROOT/Session_12" \
  --output checkpoints/easycom_frozen_states_sparse_supervision_50ep.pth \
  --epochs 50 \
  --batch-size 512 \
  --expected-mode any \
  --device cuda:0

Evaluation remains dense 5 Hz on the official test sessions:

CUDA_VISIBLE_DEVICES=3 \
python infer_turn_head.py \
  --feature-root "$FROZEN_FEATURE_ROOT" \
  --target-root "$TARGET_ROOT" \
  --checkpoint checkpoints/easycom_frozen_states_sparse_supervision_50ep.pth \
  --output-dir results/easycom_frozen_states_sparse_supervision_50ep \
  --expected-mode any \
  --device cuda:0

Compare this result against a head trained for the same 50 epochs on every dense 5 Hz anchor from the same FROZEN_FEATURE_ROOT. A large drop isolates the problem to sparse supervision; a close result means sparse supervision is not sufficient to explain the LoRA regression.

Thinker-LoRA sampling and cache policy

This is an implemented second-stage experiment, not yet a reported benchmark result. Dense 5 Hz partial-prefix caching is fast during training but does not fit the combined EasyCom and Ego4D storage budget. The 24-second Session 7 pilot used 0.115 GiB, which extrapolates to about 17.25 GiB per media hour. In the latest controlled single-L20 test without gradient checkpointing, 200 updates took 57.718 s (3.465 update/s) with every partial cached, 130.151 s (1.537 update/s) with current partials recomputed using processor defaults, and 111.146 s (1.799 update/s) after CUDA log-Mel and bounded-audio-padding optimization. The optimized online run spent 53.408 s in 163 frozen-AV calls; the cached run therefore remains the preferred training path when storage is available.

The proposed training sampler is therefore:

1. retain a uniform low-rate set of ordinary anchors, initially 1 Hz;
2. retain every 5 Hz anchor within a two-second band around annotated speaker onset, offset, handover, and overlap-state transitions;
3. optionally add high-loss or high-uncertainty anchors after a first training pass, while selecting them only from the training split;
4. cache exact partial AV embeddings only for these selected anchors, plus one reusable complete embedding for every required 2-second history block;
5. enforce a 240 GiB cache target and a hard 300 GB upper bound, leaving room for manifests, checkpoints, and extracted evaluation states.

Transition-centered selection is training-example sampling, not inference look-ahead: model inputs must still end at the current time. Final extraction, head training, and Table 2 evaluation remain on the complete 5 Hz timeline. The paper must report the sampling rule and compare uniform-only sampling with transition-enhanced sampling so that any gain is not attributed solely to additional transition examples.

cache_omni_av_pieces.py --anchor-policy transition-sparse implements this policy without changing the trainer cache schema. A transition is any change in the three-class EasyCom target vector. The default two-second dense band is defined explicitly as one second before through one second after that target row. Every complete two-second source block is still cached once as reusable history. Ordinary selected anchors and transition-band anchors cache only their exact current causal phase. Label information selects training examples offline but never changes the media endpoint supplied to Qwen.

Inspect the selection counts for the 24-second pilot without loading media or the model:

python cache_omni_av_pieces.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_7 \
  --target /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame/Session_7.npy \
  --output-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_7_24s \
  --end 24 \
  --anchor-policy transition-sparse \
  --ordinary-anchor-hz 1 \
  --transition-before-seconds 1 \
  --transition-after-seconds 1 \
  --selection-only

After auditing the counts, remove --selection-only and add the normal AV extraction options. Use --end 0 only after the 24-second cache and replay verification pass; zero selects the complete target timeline. The resulting manifest contains only selected training anchors, so train_omni_turn_lora.py uses the sparse dataset automatically and must be run without --online-current-tail.

Multi-GPU Thinker-LoRA training

Fully cached AV training supports native single-node DDP through torchrun. Each rank holds one complete 3B Thinker replica, reads a disjoint shard of the selected cache anchors, and synchronizes only the trainable LoRA and turn-head gradients. Model checkpoint loading is deliberately staggered by rank so four processes do not read all checkpoint shards simultaneously. During gradient accumulation, DDP synchronization occurs only on the final microbatch. Only rank zero writes the adapter, head, summary, and progress log.

For four GPUs, use per-GPU batch 2 and accumulation 2 for global batch 16:

CUDA_VISIBLE_DEVICES=0,1,2,3 \
OMP_NUM_THREADS=2 \
MKL_NUM_THREADS=2 \
OPENBLAS_NUM_THREADS=2 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
torchrun --standalone --nproc_per_node=4 train_omni_turn_lora.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_7 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_11 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_10 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_6 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_9 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_4 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_8 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_5 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_12 \
  --output-dir checkpoints/omni_turn_lora_easycom_sparse_8s_ddp4 \
  --max-steps 10170 \
  --batch-size 2 \
  --gradient-accumulation-steps 2 \
  --num-workers 0 \
  --block-cache-size 32 \
  --learning-rate 2e-5 \
  --head-learning-rate 1e-4 \
  --warmup-steps 500 \
  --dtype bfloat16 \
  --flash-attention-2 \
  --log-every 20

10170 is only the current three-epoch estimate for roughly 54,000 selected anchors. Recompute it from the completed manifests as ceil(num_examples / 16) * 3. Do not pass --device: under torchrun, each process uses its LOCAL_RANK. DDP intentionally rejects --online-current-tail; the online producer-consumer path remains a separate benchmark.

Dense low-drift LoRA probe

The first follow-up to the sparse-LoRA regression should remove sparse supervision as a variable and constrain adaptation near the Thinker output. Use dense 5-Hz caches, reserve Session_12 as development data, and train on Session_4 through Session_11. --lora-last-n-layers 8 discovers the language-layer indices and passes the 24 selected full module paths directly to PEFT for the 36-layer Qwen2.5-Omni-3B Thinker. Explicit paths are required because the extracted Thinker exposes names beginning with layers.0..., which some PEFT layers_pattern versions fail to recognize. Combining it with --lora-target-modules q_proj,v_proj,o_proj reduces the adapter from all 252 attention/MLP projections to 24 output-side attention projections.

Start with the successful frozen 8-second head fixed. --freeze-head requires --head-checkpoint, excludes the head from DDP and AdamW, but still backpropagates through it into the Thinker LoRA. This isolates representation adaptation from head decision-boundary drift. Do not continue from a regressed adapter: create this adapter from the original Thinker.

For four GPUs with global batch 16, compute exactly one epoch from the dense manifest anchor counts and run:

cd /data1/yfzx/wangx/Code/icassp2027-experiment

MAX_STEPS=$(python - <<'PY'
import json
import math
from pathlib import Path

root = Path("/data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s")
sessions = (4, 5, 6, 7, 8, 9, 10, 11)
examples = sum(len(json.loads((root / f"Session_{s}" / "manifest.json").read_text())["anchors"])
               for s in sessions)
print(math.ceil(examples / 16))
PY
)
WARMUP_STEPS=$(( (MAX_STEPS + 19) / 20 ))

CUDA_VISIBLE_DEVICES=0,1,2,3 \
OMP_NUM_THREADS=2 \
MKL_NUM_THREADS=2 \
OPENBLAS_NUM_THREADS=2 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
torchrun --standalone --nproc_per_node=4 train_omni_turn_lora.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_4 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_5 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_6 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_7 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_8 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_9 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_10 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_11 \
  --head-checkpoint checkpoints/easycom_open_stream_2fps_8s_head_seed0.pth \
  --freeze-head \
  --lora-target-modules q_proj,v_proj,o_proj \
  --lora-last-n-layers 8 \
  --lora-rank 8 \
  --lora-alpha 16 \
  --learning-rate 5e-6 \
  --output-dir checkpoints/omni_turn_lora_dense_8s_last8_qvo_frozen_head_epoch1 \
  --max-steps "${MAX_STEPS}" \
  --batch-size 2 \
  --gradient-accumulation-steps 2 \
  --num-workers 0 \
  --block-cache-size 32 \
  --warmup-steps "${WARMUP_STEPS}" \
  --dtype bfloat16 \
  --flash-attention-2 \
  --log-every 20

Select this recipe on dense Session_12 using 0.2-second mAP as the primary metric and mean mAP as the tie-breaker. Only after the recipe is fixed should it be evaluated on official test Sessions_1--3. A second controlled run may omit --freeze-head and use --head-learning-rate 1e-5; all other settings must remain identical.

For a continuous five-epoch run, set TOTAL_STEPS=$((5 * MAX_STEPS)) and pass --max-steps "${TOTAL_STEPS}" --save-every-steps "${MAX_STEPS}". The trainer keeps one AdamW/cosine-scheduler trajectory and writes evaluation-ready adapter/ plus turn_head.pt pairs under checkpoint-step-MAX_STEPS, checkpoint-step-2*MAX_STEPS, through the fifth epoch. The top-level adapter and head duplicate the final fifth checkpoint. Evaluate all five intermediate checkpoints only on held-out Session_12, select one, then run that checkpoint once on Sessions_1--3.

Frozen-Thinker sparse head-only control

To determine whether the regression comes from transition-sparse sampling or from updating the Thinker representation, rerun the same sparse-anchor training with --freeze-thinker. This still forwards every reconstructed 8-second sequence through the original Thinker, but runs that forward under no_grad and updates only the turn head initialized from the frozen 8-second checkpoint. It does not create an adapter directory.

For a schedule-matched comparison with a one-epoch LoRA run at global batch 16, use the same cache directories, head learning rate, warmup, seed, and approximately ceil(num_examples / 16) updates:

CUDA_VISIBLE_DEVICES=0,1,2,3 \
OMP_NUM_THREADS=2 \
MKL_NUM_THREADS=2 \
OPENBLAS_NUM_THREADS=2 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
torchrun --standalone --nproc_per_node=4 train_omni_turn_lora.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_7 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_11 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_10 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_6 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_9 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_4 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_8 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_5 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_sparse_8s/Session_12 \
  --freeze-thinker \
  --head-checkpoint checkpoints/easycom_open_stream_2fps_8s_head_seed0.pth \
  --output-dir checkpoints/easycom_sparse_head_only_from_frozen_8s_ddp4_epoch1 \
  --max-steps 3390 \
  --batch-size 2 \
  --gradient-accumulation-steps 2 \
  --num-workers 0 \
  --block-cache-size 32 \
  --head-learning-rate 1e-4 \
  --warmup-steps 500 \
  --dtype bfloat16 \
  --flash-attention-2 \
  --log-every 20

Evaluate the resulting head on the same dense test caches without an adapter:

CUDA_VISIBLE_DEVICES=0,1,2 \
OMP_NUM_THREADS=1 \
MKL_NUM_THREADS=1 \
OPENBLAS_NUM_THREADS=1 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
TORCH_NCCL_ASYNC_ERROR_HANDLING=1 \
torchrun --standalone --nproc_per_node=3 eval_omni_turn_lora.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_3 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_2 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_1 \
  --head-checkpoint checkpoints/easycom_sparse_head_only_from_frozen_8s_ddp4_epoch1/turn_head.pt \
  --output-dir results/easycom_sparse_head_only_dense_eval_epoch1 \
  --batch-size 2 \
  --block-cache-size 32 \
  --dtype bfloat16 \
  --flash-attention-2 \
  --ddp-timeout-minutes 60

If this head-only control also falls well below the frozen 58.16 dense-cache control, sparse sampling/head optimization is sufficient to explain the regression. If it stays near 58.16 while LoRA falls, the main failure is Thinker representation drift.

To continue from a completed run, pass both its adapter/ directory through --adapter-checkpoint and its turn_head.pt through --head-checkpoint while writing to a new output directory. This is weights-only continuation: the adapter and head weights resume, but AdamW moments and the cosine scheduler are reinitialized. Preserve the previous global batch when changing GPU count; for example, change 8 GPUs with batch 2 and accumulation 1 to 4 GPUs with batch 2 and accumulation 2. Use zero warmup and a reduced learning rate for a short post-hoc continuation because a one-epoch cosine schedule has already reached zero.

Evaluate a Thinker-LoRA checkpoint

Use eval_omni_turn_lora.py for the formal EasyCom Table 2 evaluation. It loads the trained Thinker LoRA and 10 x 3 turn head, reconstructs every causal state from dense AV caches, and reports both the EgoSpeak full-timeline and strict causal-valid protocols. Sparse training caches are rejected because they do not contain predictions at every 200 ms timestamp.

The following command evaluates the official EasyCom test sessions on three GPUs, with one complete session assigned to each rank:

CHECKPOINT=/data1/yfzx/wangx/Code/icassp2027-experiment/checkpoints/omni_turn_lora_easycom_sparse_8s_ddp8_epoch1

CUDA_VISIBLE_DEVICES=0,1,2 \
OMP_NUM_THREADS=1 \
MKL_NUM_THREADS=1 \
OPENBLAS_NUM_THREADS=1 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
TORCH_NCCL_ASYNC_ERROR_HANDLING=1 \
torchrun --standalone --nproc_per_node=3 eval_omni_turn_lora.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_3 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_2 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_1 \
  --adapter-checkpoint "${CHECKPOINT}/adapter" \
  --head-checkpoint "${CHECKPOINT}/turn_head.pt" \
  --output-dir results/omni_turn_lora_easycom_sparse_8s_ddp8_epoch1 \
  --batch-size 2 \
  --block-cache-size 32 \
  --dtype bfloat16 \
  --flash-attention-2 \
  --ddp-timeout-minutes 60

The evaluator saves per-session probabilities and valid masks under pred/ and valid/, and writes the aggregate metrics to metrics.json. Add --overwrite only when intentionally replacing an existing result directory. Treat Sessions 1, 2, and 3 as the official test set: inspect their metrics only after the training recipe and hyperparameters have been fixed.

To isolate whether a score change comes from LoRA or from the dense-cache reconstruction path, run the same evaluator without --adapter-checkpoint and load the original frozen 8-second head:

FROZEN_HEAD=checkpoints/easycom_open_stream_2fps_8s_head_seed0.pth

CUDA_VISIBLE_DEVICES=0,1,2 \
OMP_NUM_THREADS=1 \
MKL_NUM_THREADS=1 \
OPENBLAS_NUM_THREADS=1 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
TORCH_NCCL_ASYNC_ERROR_HANDLING=1 \
torchrun --standalone --nproc_per_node=3 eval_omni_turn_lora.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_3 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_2 \
  --cache-dir /data1/yfzx/wangx/Datasets/easycom/omni_lora_cache_dense_8s/Session_1 \
  --head-checkpoint "${FROZEN_HEAD}" \
  --output-dir results/easycom_frozen_8s_dense_cache_control \
  --batch-size 2 \
  --block-cache-size 32 \
  --dtype bfloat16 \
  --flash-attention-2 \
  --ddp-timeout-minutes 60

This control should approach the previously reported frozen 8-second score of 58.51937 full-timeline mean mAP. A material mismatch indicates a representation difference between dense-cache reconstruction and the earlier open-stream feature files; a close match isolates the regression to LoRA/head training.

Recorded sparse-LoRA results

The current single-run EasyCom results confirm negative transfer from this sparse LoRA recipe:

Checkpoint	Full-timeline mean mAP	Strict causal mean mAP
Frozen Thinker dense-cache control	58.16091	58.17146
Frozen states + sparse supervision, 50 epochs	57.20379	57.21367
Sparse LoRA epoch1	56.52480	56.53450
Frozen-head initialized LoRA, 3 epochs	56.19608	56.20535
Sparse LoRA weights-only epoch2	54.19850	54.20711

Epoch2 is a weights-only continuation: adapter/head weights resume, but AdamW and the scheduler restart. It is 2.32630 mAP points below epoch1 and 3.96241 below the frozen dense-cache control in the full-timeline protocol. The largest regression is camera-wearer speaking. Do not continue the same recipe to a third run. The complete snapshot and per-class metrics are archived in results/easycom_omni_lora_sparse_epoch2_2026-08-24/.

The frozen-state sparse-supervision and frozen-head initialized three-epoch LoRA tables are archived in results/easycom_sparse_supervision_vs_lora_2026-08-24/. At 0.2 seconds, sparse supervision alone changes mAP from 71.69695 to 71.69194 (effectively unchanged), while the three-epoch LoRA result is 71.08589. This isolates the short-horizon regression primarily to Thinker updating rather than sparse target selection.

Dense last-8 LoRA result

Restricting LoRA to q_proj, v_proj, and o_proj in Thinker layers 28--35, reducing the learning rate to 5e-6, using dense 5-Hz supervision, and freezing the trained 8-second head substantially reduces but does not reverse negative transfer. Five continuous epochs were selected on Session 12 by 0.2-second mAP; epoch 3 / step 11910 was then evaluated once on Sessions 3, 2, and 1.

Checkpoint	0.2s full mAP	Full mean mAP	Delta from frozen dense mean
Frozen Thinker dense-cache control	71.69695	58.16091	—
Dense last-8 q/v/o LoRA, frozen head, epoch 1, train 4--12	71.62100	58.03491	-0.12600
Dense last-8 q/v/o LoRA, frozen head, epoch 1, train 4--11	71.59585	58.01500	-0.14591
Dense last-8 q/v/o LoRA, frozen head, epoch 3, train 4--11	71.65000	57.88833	-0.27258
Prior frozen-head initialized sparse LoRA, 3 epochs	71.08589	56.19608	-1.96483

Epoch 1 is only 0.05415 below epoch 3 at 0.2 seconds but is 0.12667 higher in mean mAP. It reduces the mean regression relative to frozen dense from -0.27258 to -0.14591, with better results than epoch 3 at every horizon from 0.6 through 2.0 seconds. This supports fixing one epoch and retraining on all official training Sessions 4--12 rather than selecting a later checkpoint. That full-train run reaches 71.62100 at 0.2 seconds and 58.03491 mean, improving the Sessions 4--11 epoch-1 result by only 0.02515 and 0.01991. It remains 0.12600 mean points below frozen dense, confirming that withholding Session 12 was not the cause of the residual negative transfer. Its complete record is in results/easycom_dense_last8_qvo_full_train_epoch1_2026-08-25/. For epoch 3, the gap grows almost monotonically to -0.45040 at 2.0 seconds and camera-wearer AP is the main regression (-0.62718 average). The complete development table, per-horizon/class test table, configuration, and provenance caveats are archived in results/easycom_dense_last8_qvo_frozen_head_epoch3_2026-08-25/.

This result should be reported as a negative-transfer ablation, not as an Omni improvement. Any subsequent adapter interpolation or objective change must be chosen without consulting Sessions 1--3; for a clean publication protocol, the frozen head must also be retrained without the chosen development session.

For a speed-only alternative, train_omni_turn_lora.py supports a two-GPU producer-consumer benchmark. The producer GPU runs the frozen AV towers and places current-tail embeddings in a bounded CPU queue while the consumer GPU runs Thinker LoRA. This preserves exact online partial semantics; it changes only scheduling. Compare its reported update rate and consumer queue-wait time with the synchronous 0.787 update/s pilot before adopting it.

EgoSpeak-compatible evaluation

Run inference on the official test sessions (Session_3, Session_2, and Session_1):

python infer_turn_head.py \
  --feature-root features/open_2fps \
  --target-root target_frame \
  --checkpoint checkpoints/open_2fps_head.pth \
  --output-dir results/open_2fps_head \
  --device cuda:0

This writes pred/Session_X.npy with the exact LSTR/EgoSpeak layout [target_time, class, future_horizon], i.e. [T,3,10], plus metrics.json. For every valid current timestep, the head itself produces probabilities shaped:

[number of timesteps, 10 future horizons, 3 classes]

Then reproduce EgoSpeak's Table 2 procedure:

1. for each horizon independently, concatenate test-session predictions and ground truth;
2. compute per-class frame-level average precision;
3. average the three class AP values to obtain horizon mAP;
4. report all ten horizon mAP values;
5. average the ten horizon mAP values for the final score.

With EgoSpeak's ignore_index=-100, the released evaluator does not remove class index zero, so background is included. The script reports two protocols:

• egospeak_full_timeline: the same full [T,3,10] array and evaluator as LSTR, with zeros where a prediction would require pre-session context;
• causal_valid_only: excludes only those unavailable first h rows for horizon h. This is the primary strictly causal score.

The difference involves at most the first ten rows of each session, but both are saved so the comparison is explicit.

The convenience wrapper mirrors EgoSpeak's train/inference shells:

chmod +x run_easycom_head_experiment.sh
CUDA_VISIBLE_DEVICES=0 ./run_easycom_head_experiment.sh all

Reference Table 2 targets from the EgoSpeak paper:

• EasyCom GRU A+V: 60.6 mean mAP;
• EasyCom Transformer A+V: 58.7 mean mAP;
• Ego4D Transformer audio: 69.2 mean mAP;
• Ego4D Transformer A+V: 69.0 mean mAP;
• Ego4D GRU audio: 69.2 mean mAP;
• Ego4D GRU A+V: 68.2 mean mAP.

Do not compare these numbers with EgoSpeak Table 3 target-speaker-only AP; Table 3 uses a different metric.

Current paper experiment ladder

The headline comparison cannot establish speaker grounding or social understanding. Experiments therefore proceed through explicit gates:

1. P0 — protocol correctness: preserve physical 5 Hz timing and causal masks. The evaluator now prevents an other-offset response from being attributed at/after a new other onset and symmetrically censors yield at the next other offset; run the real-data audit and freeze scenario rules without tuning on EasyCom test Sessions 1--3.
2. P1 — representation diagnosis: compare EgoSpeak features, Qwen encoder-only A/V/AV, the global Thinker state, and anonymous person slots on aggregate activity and per-person active-speaker probes. Include wrong-person, time-shifted, visual-only, and audio-only controls.
3. P2 — behavioral baselines: evaluate G, G+A, G+A+pooled-V, shuffled visual features, and a parameter/history-matched temporal control using entry/yield event AP and third-party false entries at matched onset recall.
4. P3 — primary method: use a shared causal person encoder plus wearer-state-conditioned attention, then ablate per-person active-speaker, addressee, and wearer event supervision.
5. P4 — Omni adaptation ceiling: compare parameter-matched Thinker LoRA, local person-memory adapters, and adapter+LoRA only if person features pass P1 and improve floor behavior in P3.

Original 0.2--2.0-second EgoSpeak AP/mAP remains a secondary comparability metric. The primary paper evidence is selective behavior: correct person alignment must reduce false wearer entry during third-party conversation while preserving addressed response and yield. Full hypotheses, tables, stopping rules, and the 8×L20 execution order are in the current experiment plan.

Secondary semantic analyses can still include context/layer ablations, completed-versus-hesitation subsets, question/statement/backchannel subsets, and a causal ASR-text baseline. They support interpretation but cannot replace the person-alignment interventions above.

Transcript-derived interaction-event proxies

EasyCom's released human transcriptions contain inclusive 20 FPS Start_Frame/End_Frame, Participant_ID, Transcription, and Target_of_Speech. They do not contain a semantic backchannel class. derive_easycom_interaction_events.py therefore separates observed timing relations from weak semantic proxies:

• ordinary_switch: a non-overlapping recent speaker change followed by a sustained incoming turn;
• overlap_takeover: an observed overlapping onset followed by sustained takeover, without claiming interruption intent;
• overlap_hold: overlap without enough evidence for takeover;
• backchannel_proxy_high: a conservative short lexical response while the same other participant dominates immediately before and after it;
• interruption_proxy_high: the previous transcript has a non-boundary trailing cutoff and the incoming participant takes over;
• nonverbal_feedback: laughter-only feedback around another participant's floor; pure [H], [C], [T], and [U] records are ambiguous candidates.

backchannel_candidate, interruption_candidate, and nonverbal_candidate are retained for auditing but masked from auxiliary training. [X] continuations across adjacent one-minute files are not new events. Source-time gaps larger than one second create separate segments, so the dense EgoSpeak concatenation does not create a false switch across a redacted interval. A one- or two-frame annotation/media endpoint discrepancy is clamped and reported; larger disagreement is fatal.

A read-only audit on the raw-data machine is:

cd /mnt/58462F4C462F29E8/Code/icassp2027-experiment

python3 derive_easycom_interaction_events.py \
  --dataset-root /mnt/58462F4C462F29E8/Datasets/easycom \
  --target-root /mnt/58462F4C462F29E8/Code/EgoSpeak/dataset/EasyCom/target_perframe \
  --examples-per-event 10

To persist JSONL/CSV reports outside the dataset, add:

  --output-dir reports/easycom_interaction_events_v1 \
  --overwrite

In an environment with NumPy, add --write-frame-arrays to also write one Session_*.aux_targets.npz per session. It contains 5 Hz event_onsets, event_roles, collision flags, and event-name metadata aligned to target_perframe. The raw EasyCom tree is never modified.

The initial all-session audit found 6,981 utterances, including 530 high-confidence backchannel proxies, 78 high-confidence interruption proxies, 937 overlap-takeovers, 686 overlap-holds, and 349 laughter-feedback events. These counts establish usable weak-supervision coverage, not label accuracy. Before training, inspect a stratified sample from Sessions 4--12 and freeze the rules; do not tune thresholds on test Sessions 1--3. The primary evaluator remains the original EgoSpeak three-label AP/mAP protocol.

Optional DeepSeek adjudication of ambiguous candidates

DeepSeek is not required for the deterministic timing labels above. It is an optional second-stage weak-label adjudicator for only backchannel_candidate and interruption_candidate; sending all 6,981 utterances to an LLM would add cost and noise without adding observable prosody, gaze, or intent. The current audit contains 614 such candidates over all sessions and 497 in the training split (Sessions 4--12).

adjudicate_easycom_interaction_events.py uses DeepSeek JSON Output through the standard-library HTTP client, so no additional Python package is required. It makes two differently instructed conservative passes. A candidate is promoted only if both passes return the candidate's matching semantic label with high confidence; disagreements, medium/low confidence, short answers, cooperative overlaps, and uncertain cases remain masked. The result is still a weak proxy and must not be presented as human ground truth. The append-only JSONL records prompt/model metadata and resumes completed candidates, while rejecting a mixed model or prompt configuration.

First inspect exactly what would leave the machine. This makes no API call and does not require a key:

python3 adjudicate_easycom_interaction_events.py \
  --input-dir reports/easycom_interaction_events_v1 \
  --output reports/easycom_interaction_llm_v1.jsonl \
  --max-events 16 \
  --dry-run

If external transcript upload is authorized, enter the key without placing it in shell history and run a 32-candidate pilot:

read -s -p "DeepSeek API key: " DEEPSEEK_API_KEY; echo
export DEEPSEEK_API_KEY

python3 adjudicate_easycom_interaction_events.py \
  --input-dir reports/easycom_interaction_events_v1 \
  --output reports/easycom_interaction_llm_pilot_v1.jsonl \
  --max-events 32 \
  --selection-seed 2027 \
  --batch-size 8 \
  --passes 2 \
  --allow-external-transcript-upload

unset DEEPSEEK_API_KEY

Audit the pilot rationales and precision manually before running the full training split. For the full pass, use a new output path, remove --max-events, and keep the default --splits train. The default model is deepseek-v4-flash; it can be changed with --model or DEEPSEEK_MODEL as the provider's available model names evolve. Never tune the prompt or acceptance rule using Sessions 1--3. LLM-adjudicated candidates are deliberately kept in a separate JSONL until pilot precision is accepted; they are not yet silently merged into the training arrays.

Anonymous person-track GRU residual

The first person-temporal experiment is implemented as --local-fusion track-gru-residual. It preserves an exactly initialized G+A+pooled-V head and learns a zero-gated correction from shared causal GRU states over temporary person tracks. IDs route history only; they are not model inputs. Source-clip boundaries, gaps over the configured maximum and unknown IDs reset history. Existing local Qwen sidecars can be reused without re-extraction.

Run the matched pooled/track development protocol with:

CUDA_VISIBLE_DEVICES=0 bash run_easycom_track_gru_residual.sh all 0

Use separate GPUs for additional seeds, then summarize the common result root with summarize_local_baselines.py --paired-deltas. After method selection is frozen, bash run_easycom_track_gru_residual.sh test 0 1 2 evaluates the saved checkpoints once on Sessions 1--3. The exact causal contract, EgoSpeak code comparison, L20 smoke command and stopping rule are in docs/easycom_track_gru_residual.md. This is a frozen-backbone side branch, not an internal Thinker modification or an EgoSpeak MiniROAD reproduction.

Candidate improvement: explicit working and long memory

Motivated by EgoSpeak's separation of long-term and working memory, test an explicit two-timescale representation instead of compressing the entire causal window into one undifferentiated final Thinker state. At prediction time t, align long memory to fixed global 2-second boundaries and let working memory slide with the current prediction time:

b = floor(t / 2 s) * 2 s
long memory:     complete 2-second chunks ending at b, ..., [b-4, b-2], [b-2, b]
working memory:  the sliding recent interval [t-2, t], updated every 0.2 s

For example, at t = 20.4 s, long memory contains the fixed chunks [16,18], [18,20], and any earlier retained chunks, while the working partial is [18.4,20.4]. The overlap between [18,20] and [18.4,20.4] is intentional: long memory provides a stable completed-chunk representation, whereas working memory preserves the exact high-resolution context ending at the current prediction time. Cache one representation per completed long-memory chunk, recompute the working-memory representation at each 5 Hz prediction step, and fuse the two branches with a small gated/attention pooling module before the same 10 x 3 prediction head. Keep the Omni backbone frozen in the first test so that any gain can be attributed to memory organization rather than additional backbone capacity.

This is a candidate score-improvement idea, not yet an experimental result. The key hypothesis is that a dedicated recent branch preserves local prosody, speaker-change, and utterance-completion cues while long memory supplies conversation context without diluting them in one final state. Evaluate it against matched-compute baselines: (1) the current single-state 8-second model, (2) working-memory-only [t-2,t], and (3) long+working memory with the same total causal context. Report parameter count, throughput, and per-horizon and per-class AP over identical seeds. A useful result requires the fused model to beat both controls consistently; otherwise a gain could merely come from the extra fusion parameters or the stronger recent-window shortcut.

Frozen serialized-recap ablation (implemented)

Before adding a learned two-branch fusion module, test a stricter frozen-Omni variant with exactly the same 2048-dimensional readout and prediction head as the current 8-second baseline. Mode overlap-working-memory serializes the two timescales inside one still-open Thinker media turn:

total AV presentation budget = 8 s
fixed history budget         = 6 s = three cached 2 s blocks
working recap                = 2 s = [t-2,t]

t = 20.4 s:
fixed:   [14,16] [16,18] [18,20]
working:                   [18.4,20.4]
                                      -> one frozen Thinker state [2048]

The overlap is intentional. It presents recent evidence twice but does not add AV-token duration beyond the matched 8-second budget. Thus this ablation trades older unique context for stronger recent evidence instead of silently giving the candidate more input tokens. Fixed blocks retain the global half-open 2 FPS grid. The working recap reuses three causal global-grid history frames and replaces the oldest visual slot with the current causal frame:

fixed [18,20):       18.0  18.5  19.0  19.5
working [18.4,20.4]: 18.5  19.0  19.5  20.4

Keeping four frames means the oldest visual tick is replaced by the current frame rather than adding a fifth frame; this isolates endpoint recency without increasing visual tokens. Historical target times are causally floored to the session-global 2 FPS grid. This lets them share the complete-block frame cache and avoids making Decord seek backward through a long-GOP H.264 stream for four new endpoint-shifted timestamps every 200 ms. Audio remains the full, exactly half-open [start,end) slice. Consequently, call this a global-history plus current-frame recent-video intervention, not an exact timestamp-preserving resampling of both modalities. The recap is serialized as the next TMRoPE chunk; it is deliberately a recent-evidence recap, not a claim that duplicated samples occur later in physical time. This mode is an ablation, not native Qwen streaming semantics, and the Omni backbone stays frozen.

One deliberate side effect is a 2-second phase cycle: overlap falls toward 0.2 seconds just before a fixed boundary and jumps to 2.0 seconds at the boundary. Every output stores this value in overlap_working_memory_policy.layout. If aggregate mAP changes, also inspect performance by step_index % 10; a boundary-only gain or failure is a chunk phase artifact rather than general evidence for better recent semantics.

Run a 24-second L20 smoke test first:

The sliding 2-second recap otherwise invokes Whisper's model-maximum audio padding 120 times. Run log-Mel extraction on the model GPU and set explicit right padding to zero. Every EasyCom 0.2-second increment contains 3200 samples at 16 kHz and is already divisible by Whisper's 160-sample hop, so no external silence is needed. This preserves the feature extractor's own STFT boundary handling while avoiding repeated processing of tens of seconds of zeros.

mkdir -p features/easycom_overlap_working_8s_smoke

CUDA_VISIBLE_DEVICES=3 \
OMP_NUM_THREADS=4 \
MKL_NUM_THREADS=4 \
OPENBLAS_NUM_THREADS=4 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
python streaming_turn_model_directly.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_7 \
  --end 24 \
  --mode overlap-working-memory \
  --memory-seconds 8 \
  --working-memory-seconds 2 \
  --video-fps 2 \
  --frame-cache-size 128 \
  --audio-feature-device model \
  --audio-right-padding-samples 0 \
  --decoder decord \
  --decoder-threads 1 \
  --max-open-readers 1 \
  --dtype bfloat16 \
  --flash-attention-2 \
  --device cuda:0 \
  --verify-every 0 \
  --output features/easycom_overlap_working_8s_smoke/Session_7_24s_global_history_current.pt

Expected shape is (120, 2048). This speed smoke test disables replay verification; after its throughput is acceptable, rerun a short four-second check with --verify-every 5, where overlap_cache_replay should be numerically exact or effectively exact. Inspect the saved timing policy and decoder cache counters with:

python - <<'PY'
import torch

p = torch.load(
    "features/easycom_overlap_working_8s_smoke/Session_7_24s_global_history_current.pt",
    map_location="cpu",
    weights_only=False,
)
print(p["mode"], tuple(p["states"].shape))
print(p["overlap_working_memory_policy"]["working_media"])
print(p["direct_media"]["runtime_counters"])
for row in p["overlap_working_memory_policy"]["layout"][98:102]:
    print(row)
PY

For one complete session, replace --end 24 with its target and use a session-level output filename:

mkdir -p /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_global_history_current_2fps_8s

CUDA_VISIBLE_DEVICES=3 \
OMP_NUM_THREADS=4 \
MKL_NUM_THREADS=4 \
OPENBLAS_NUM_THREADS=4 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
python streaming_turn_model_directly.py \
  --model-id /data1/yfzx/wangx/Models/Qwen2.5-Omni-3B \
  --easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_7 \
  --target /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame/Session_7.npy \
  --mode overlap-working-memory \
  --memory-seconds 8 \
  --working-memory-seconds 2 \
  --video-fps 2 \
  --frame-cache-size 128 \
  --audio-feature-device model \
  --audio-right-padding-samples 0 \
  --decoder decord \
  --decoder-threads 1 \
  --max-open-readers 1 \
  --dtype bfloat16 \
  --flash-attention-2 \
  --device cuda:0 \
  --output /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_global_history_current_2fps_8s/Session_7.pt

After all train/test sessions have been extracted, train the unchanged frozen head by setting only the expected feature mode:

python train_turn_head.py \
  --feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_global_history_current_2fps_8s \
  --target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
  --output checkpoints/easycom_overlap_working_frozen_8s.pt \
  --expected-mode overlap-working-memory \
  --epochs 50 \
  --batch-size 512 \
  --device cuda:0

python infer_turn_head.py \
  --feature-root /data1/yfzx/wangx/Datasets/easycom/omni_features/overlap_global_history_current_2fps_8s \
  --target-root /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame \
  --checkpoint checkpoints/easycom_overlap_working_frozen_8s.pt \
  --output-dir results/easycom_overlap_working_frozen_8s \
  --expected-mode overlap-working-memory \
  --device cuda:0

The decisive comparison is against the frozen single-state 8-second baseline using the same split, head, optimization, and seeds. Report the full per-horizon curve: the hypothesis predicts the clearest gain at 0.2--1.0 seconds; degradation at 1.2--2.0 seconds would indicate that repeated local cues displaced too much useful long context. Only if this parameter-matched test is positive should the learned two-branch fusion experiment be added.

Single-window feature inspection

qwen_omni_features.py is useful for inspecting a short window or layer, but it is not the main whole-session experiment.

If audio is embedded in the video:

python qwen_omni_features.py \
  --video /path/to/window.mp4 \
  --use-audio-in-video \
  --output features/window.pt

If audio and video are separate:

python qwen_omni_features.py \
  --video /path/to/window.mp4 \
  --audio /path/to/window.wav \
  --output features/window.pt

Do not provide the same audio both as a separate file and via --use-audio-in-video.

Saved single-window fields are:

• token_features: [B, token_length, 2048];
• last_token: [B, 2048];
• tail_mean: [B, 2048];
• attention_mask: valid-token mask.

Installation

Install ffmpeg, then:

pip install -r requirements.txt

The implementation is pinned to transformers==4.52.3 because the open-stream replay path uses version-specific Thinker internals. On Linux, qwen-omni-utils[decord] is recommended for faster video decoding. On a compatible NVIDIA GPU, optionally install flash-attn and pass --flash-attention-2.

The separate B1 local extractor, extract_turn_local_features.py, is tested with both Transformers 4.52.3 and 4.57.6. An existing 4.57.6 GPU container does not need to be downgraded for local extraction. This does not change the replay API's dependency assumptions. Keep all local sidecars for one experiment on the same version; their metadata records the actual runtime version. See the local feature guide.

Ego4D Audio-Visual Diarization download

Do not download the complete Ego4D release for this experiment. Download only the compressed Audio-Visual Diarization benchmark clips and annotations:

ego4d \
  --output_directory=/path/to/ego4d \
  --datasets clips annotations \
  --benchmarks AV

The approximately 1 TB figure on the Ego4D download page refers to all benchmark clips combined, not only the AV subset. The annotated AVD subset has about 572 five-minute clips (roughly 50 hours). EgoSpeak uses a custom 346/87 train/test split, about 36.1 hours total. Reproducing its Table 2 requires the exact IDs in EgoSpeak's EGO4DSHUFFLE split.

Required validation before a full run

Do not launch full EasyCom/Ego4D extraction until all of these pass:

• no input audio sample or video frame has timestamp later than prediction time t;
• states and target_perframe lengths/time origins are reconciled and the chosen label offset is saved;
• open-stream cache replay is exactly equivalent to its from-scratch replay on short clips;
• open-stream and causal-prefix predictions are compared on identical clips;
• a 20-second run fits the 3090 with the selected pixel budget and attention implementation;
• saved outputs contain exactly ten future horizons in the order 0.2--2.0 s;
• the local metric reproduces EgoSpeak's evaluator on known predictions;
• test session IDs never enter training, validation, normalization statistics, or hyperparameter selection.

Once these checks pass, the historical Stage-0 representation baseline is:

frozen Qwen2.5-Omni-3B Thinker
+ trained 10x3 head
+ bounded Qwen-style 2-second open streaming
+ 20-second rolling causal memory
+ full-session 5 Hz output
+ exact EgoSpeak split and Table 2 metric

The repository workflow was first verified on 2026-08-19. The paper question and gated experiment plan were revised on 2026-08-30.