144 lines
5.7 KiB
Markdown
144 lines
5.7 KiB
Markdown
# EasyCom no-box full-frame Qwen AV: F2/F3
|
||
|
||
## Definition
|
||
|
||
This experiment adds full-frame Qwen vision-encoder evidence to the existing
|
||
strictly causal global Omni plus recent Qwen-audio baseline. It never reads
|
||
face/head boxes, track IDs, active-speaker labels or speech labels.
|
||
|
||
```text
|
||
F2: G(t) + A(t) query latest full-frame Qwen spatial tokens before t
|
||
|
||
F3: same-grid-position tokens from the latest causal frames
|
||
-> shared chronological GRU
|
||
-> G(t) + A(t) queries the temporally encoded spatial grid
|
||
```
|
||
|
||
At the default 2 FPS, F3 keeps two observations per grid position: at most the
|
||
latest one second of visual history. `G` already contains global audio/video
|
||
context, so this tests whether an explicit recent encoder readout adds timely
|
||
evidence. It is not an audio-free Omni baseline.
|
||
|
||
F2/F3 are no-box full-frame visual-context models. Improvement does not by
|
||
itself establish speaker grounding, identity or addressee reasoning.
|
||
|
||
## Causal contract
|
||
|
||
`extract_fullframe_qwen_av_features.py` creates
|
||
`qwen-fullframe-av-tokens-v1` sidecars:
|
||
|
||
- a cutoff `t` sees only fixed-grid frames with timestamp strictly less than
|
||
`t`; a frame exactly at `t` is excluded;
|
||
- images are complete EasyCom camera frames after the standard session-banner
|
||
crop;
|
||
- Qwen post-merger spatial tokens are stored without mean pooling;
|
||
- F3 links only the same merged-grid row/column within one source clip, and
|
||
resets on clip boundaries, grid changes or excessive time gaps;
|
||
- audio is copied from the existing audited `[t-1,t)` Qwen mixed-audio
|
||
sidecar; its person-crop tensors are never consumed;
|
||
- visual/audio model identity, config digest and Transformers version must
|
||
match exactly;
|
||
- partial smoke sidecars are rejected by training.
|
||
|
||
The targets remain the unchanged 5 Hz `background/wearer/others` labels and
|
||
the ten 0.2--2.0-second horizons.
|
||
|
||
## 1. Four-second GPU smoke extraction
|
||
|
||
```bash
|
||
cd /data1/yfzx/wangx/Code/icassp2027-experiment
|
||
|
||
DATA_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom
|
||
MODEL=/data1/yfzx/wangx/Models/Qwen2.5-Omni-3B
|
||
FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
|
||
TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
|
||
AUDIO_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/local_qwen_1s_encoded_tf4576
|
||
SMOKE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/fullframe_qwen_av_smoke
|
||
|
||
CUDA_VISIBLE_DEVICES=0 python extract_fullframe_qwen_av_features.py \
|
||
--data-root "$DATA_ROOT" --feature-root "$FEATURE_ROOT" \
|
||
--target-root "$TARGET_ROOT" --audio-feature-root "$AUDIO_ROOT" \
|
||
--output-root "$SMOKE_ROOT" --samples 4 --max-steps 20 \
|
||
--model-id "$MODEL" --device cuda:0 --dtype bfloat16 \
|
||
--flash-attention-2 --video-fps 2 --max-pixels 50176 \
|
||
--image-batch-size 8
|
||
```
|
||
|
||
Expected: 20 states spanning 0.2--4.0 s; eight visual frames at
|
||
`0.0,...,3.5`; finite nonempty spatial tokens; matched 2048D audio; and
|
||
`partial=True`. Never train on this smoke file.
|
||
|
||
If the audio sidecars were made with an explicit `--encoder-id`, pass that
|
||
exact value here. Otherwise omit it, as above, so both branches use the same
|
||
recorded local checkpoint path.
|
||
|
||
Run the regression tests in the remote environment:
|
||
|
||
```bash
|
||
python -m unittest -v test_fullframe_qwen_av test_local_turn_features
|
||
```
|
||
|
||
They verify strict boundary exclusion, same-grid F3 history, malformed and
|
||
partial-contract rejection, and F2/F3 forward/backward behavior.
|
||
|
||
## 2. Extract all sessions
|
||
|
||
Use a fresh root. Existing files are not overwritten unless `--overwrite` is
|
||
explicitly provided.
|
||
|
||
```bash
|
||
FULLFRAME_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/fullframe_qwen_av_2fps_224
|
||
|
||
CUDA_VISIBLE_DEVICES=0 python extract_fullframe_qwen_av_features.py \
|
||
--data-root "$DATA_ROOT" --feature-root "$FEATURE_ROOT" \
|
||
--target-root "$TARGET_ROOT" --audio-feature-root "$AUDIO_ROOT" \
|
||
--output-root "$FULLFRAME_ROOT" \
|
||
--samples 1 2 3 4 5 6 7 8 9 10 11 12 \
|
||
--model-id "$MODEL" --device cuda:0 --dtype bfloat16 \
|
||
--flash-attention-2 --video-fps 2 --max-pixels 50176 \
|
||
--image-batch-size 8
|
||
```
|
||
|
||
On interruption, rerun only missing session numbers. Do not mix checkpoints,
|
||
Transformers versions, FPS settings or pixel budgets in one root.
|
||
|
||
## 3. Train and evaluate
|
||
|
||
The launcher uses Sessions 4--12 for four fixed epochs and evaluates Sessions
|
||
3/2/1 with the existing LSTR-MCE ten-horizon protocol.
|
||
|
||
```bash
|
||
export FEATURE_ROOT TARGET_ROOT FULLFRAME_ROOT
|
||
export CHECKPOINT_ROOT=checkpoints/easycom_fullframe_qwen_av
|
||
export RESULT_ROOT=results/easycom_fullframe_qwen_av
|
||
|
||
CUDA_VISIBLE_DEVICES=0 bash run_easycom_fullframe_qwen_av.sh f2 0
|
||
CUDA_VISIBLE_DEVICES=1 bash run_easycom_fullframe_qwen_av.sh f3 0
|
||
```
|
||
|
||
After seed 0 passes shape/loss/inference checks:
|
||
|
||
```bash
|
||
CUDA_VISIBLE_DEVICES=0 bash run_easycom_fullframe_qwen_av.sh f2 1 2
|
||
CUDA_VISIBLE_DEVICES=1 bash run_easycom_fullframe_qwen_av.sh f3 1 2
|
||
```
|
||
|
||
Defaults are train batch 64, inference batch 128, F3 history 2 frames and
|
||
maximum inter-frame link gap 0.75 s. Lower `TRAIN_BATCH_SIZE` or
|
||
`INFER_BATCH_SIZE` if needed; do not change the feature contract.
|
||
|
||
The checkpoint-compatible option remains named `--person-history-frames`; in
|
||
F3 it means observations per spatial grid position, not person history.
|
||
|
||
## Required comparison and claim boundary
|
||
|
||
Compare matched seeds for `G+A`, F2 and F3. Add a past time-shifted full-frame
|
||
control before attributing gains to AV synchronization. F3 contains a GRU that
|
||
F2 does not, so F3–F2 combines extra history and temporal capacity; a
|
||
parameter-matched audio/query temporal control is required before claiming a
|
||
visual-motion mechanism.
|
||
|
||
Dense AP/mAP is an architecture screen. Floor-entry/yield interpretations
|
||
remain timing-derived proxies until the competing-risk real-data audit is
|
||
frozen.
|