icassp2027-experiment/docs/easycom_fullframe_qwen_av.md

# EasyCom no-box full-frame Qwen AV: F2/F3

## Definition

This experiment adds full-frame Qwen vision-encoder evidence to the existing
strictly causal global Omni plus recent Qwen-audio baseline. It never reads
face/head boxes, track IDs, active-speaker labels or speech labels.

```text
F2: G(t) + A(t) query latest full-frame Qwen spatial tokens before t

F3: same-grid-position tokens from the latest causal frames
      -> shared chronological GRU
      -> G(t) + A(t) queries the temporally encoded spatial grid
```

At the default 2 FPS, F3 keeps two observations per grid position: at most the
latest one second of visual history. `G` already contains global audio/video
context, so this tests whether an explicit recent encoder readout adds timely
evidence. It is not an audio-free Omni baseline.

F2/F3 are no-box full-frame visual-context models. Improvement does not by
itself establish speaker grounding, identity or addressee reasoning.

## Causal contract

`extract_fullframe_qwen_av_features.py` creates
`qwen-fullframe-av-tokens-v1` sidecars:

- a cutoff `t` sees only fixed-grid frames with timestamp strictly less than
  `t`; a frame exactly at `t` is excluded;
- images are complete EasyCom camera frames after the standard session-banner
  crop;
- Qwen post-merger spatial tokens are stored without mean pooling;
- F3 links only the same merged-grid row/column within one source clip, and
  resets on clip boundaries, grid changes or excessive time gaps;
- audio is copied from the existing audited `[t-1,t)` Qwen mixed-audio
  sidecar; its person-crop tensors are never consumed;
- visual/audio model identity, config digest and Transformers version must
  match exactly;
- partial smoke sidecars are rejected by training.

The targets remain the unchanged 5 Hz `background/wearer/others` labels and
the ten 0.2--2.0-second horizons.

## 1. Four-second GPU smoke extraction

```bash
cd /data1/yfzx/wangx/Code/icassp2027-experiment

DATA_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom
MODEL=/data1/yfzx/wangx/Models/Qwen2.5-Omni-3B
FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
AUDIO_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/local_qwen_1s_encoded_tf4576
SMOKE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/fullframe_qwen_av_smoke

CUDA_VISIBLE_DEVICES=0 python extract_fullframe_qwen_av_features.py \
  --data-root "$DATA_ROOT" --feature-root "$FEATURE_ROOT" \
  --target-root "$TARGET_ROOT" --audio-feature-root "$AUDIO_ROOT" \
  --output-root "$SMOKE_ROOT" --samples 4 --max-steps 20 \
  --model-id "$MODEL" --device cuda:0 --dtype bfloat16 \
  --flash-attention-2 --video-fps 2 --max-pixels 50176 \
  --image-batch-size 8
```

Expected: 20 states spanning 0.2--4.0 s; eight visual frames at
`0.0,...,3.5`; finite nonempty spatial tokens; matched 2048D audio; and
`partial=True`. Never train on this smoke file.

If the audio sidecars were made with an explicit `--encoder-id`, pass that
exact value here. Otherwise omit it, as above, so both branches use the same
recorded local checkpoint path.

Run the regression tests in the remote environment:

```bash
python -m unittest -v test_fullframe_qwen_av test_local_turn_features
```

They verify strict boundary exclusion, same-grid F3 history, malformed and
partial-contract rejection, and F2/F3 forward/backward behavior.

## 2. Extract all sessions

Use a fresh root. Existing files are not overwritten unless `--overwrite` is
explicitly provided.

```bash
FULLFRAME_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/fullframe_qwen_av_2fps_224

CUDA_VISIBLE_DEVICES=0 python extract_fullframe_qwen_av_features.py \
  --data-root "$DATA_ROOT" --feature-root "$FEATURE_ROOT" \
  --target-root "$TARGET_ROOT" --audio-feature-root "$AUDIO_ROOT" \
  --output-root "$FULLFRAME_ROOT" \
  --samples 1 2 3 4 5 6 7 8 9 10 11 12 \
  --model-id "$MODEL" --device cuda:0 --dtype bfloat16 \
  --flash-attention-2 --video-fps 2 --max-pixels 50176 \
  --image-batch-size 8
```

On interruption, rerun only missing session numbers. Do not mix checkpoints,
Transformers versions, FPS settings or pixel budgets in one root.

## 3. Train and evaluate

The launcher uses Sessions 4--12 for four fixed epochs and evaluates Sessions
3/2/1 with the existing LSTR-MCE ten-horizon protocol.

```bash
export FEATURE_ROOT TARGET_ROOT FULLFRAME_ROOT
export CHECKPOINT_ROOT=checkpoints/easycom_fullframe_qwen_av
export RESULT_ROOT=results/easycom_fullframe_qwen_av

CUDA_VISIBLE_DEVICES=0 bash run_easycom_fullframe_qwen_av.sh f2 0
CUDA_VISIBLE_DEVICES=1 bash run_easycom_fullframe_qwen_av.sh f3 0
```

After seed 0 passes shape/loss/inference checks:

```bash
CUDA_VISIBLE_DEVICES=0 bash run_easycom_fullframe_qwen_av.sh f2 1 2
CUDA_VISIBLE_DEVICES=1 bash run_easycom_fullframe_qwen_av.sh f3 1 2
```

Defaults are train batch 64, inference batch 128, F3 history 2 frames and
maximum inter-frame link gap 0.75 s. Lower `TRAIN_BATCH_SIZE` or
`INFER_BATCH_SIZE` if needed; do not change the feature contract.

The checkpoint-compatible option remains named `--person-history-frames`; in
F3 it means observations per spatial grid position, not person history.

## Required comparison and claim boundary

Compare matched seeds for `G+A`, F2 and F3. Add a past time-shifted full-frame
control before attributing gains to AV synchronization. F3 contains a GRU that
F2 does not, so F3–F2 combines extra history and temporal capacity; a
parameter-matched audio/query temporal control is required before claiming a
visual-motion mechanism.

Dense AP/mAP is an architecture screen. Floor-entry/yield interpretations
remain timing-derived proxies until the competing-risk real-data audit is
frozen.