Merge branch 'main' of http://182.92.86.220:8300/wx44wx/icassp2027-experiment

docs/README.md

@@ -26,6 +26,7 @@ tests.
 | [`easycom_multiwindow_features.md`](easycom_multiwindow_features.md) | Shared-AV extraction of independent Qwen 2/4/6/8/10/12-second context banks, GPU smoke and matched head training |
 | [`local_feature_baseline.md`](local_feature_baseline.md) | Extracting/validating EasyCom pooled local Qwen visual and audio sidecars |
 | [`easycom_encoder_only.md`](easycom_encoder_only.md) | Comparing EasyCom Qwen encoder-only A/V/AV representations |
+| [`easycom_fullframe_qwen_av.md`](easycom_fullframe_qwen_av.md) | No-box F2/F3 full-frame Qwen spatial and causal spatiotemporal fusion |
 | [`ego4d_encoder_only.md`](ego4d_encoder_only.md) | Preparing and running the Ego4D encoder-only comparison |
 | [`ego4d_local_baselines.md`](ego4d_local_baselines.md) | Previewing Ego4D boxes, extracting local features and running 8-GPU jobs |
 | [`easycom_track_gru_residual.md`](easycom_track_gru_residual.md) | Matched pooled-to-track GRU experiment, causal identity contract and L20 commands |

docs/easycom_fullframe_qwen_av.md

@@ -0,0 +1,143 @@
+# EasyCom no-box full-frame Qwen AV: F2/F3
+
+## Definition
+
+This experiment adds full-frame Qwen vision-encoder evidence to the existing
+strictly causal global Omni plus recent Qwen-audio baseline. It never reads
+face/head boxes, track IDs, active-speaker labels or speech labels.
+
+```text
+F2: G(t) + A(t) query latest full-frame Qwen spatial tokens before t
+
+F3: same-grid-position tokens from the latest causal frames
+      -> shared chronological GRU
+      -> G(t) + A(t) queries the temporally encoded spatial grid
+```
+
+At the default 2 FPS, F3 keeps two observations per grid position: at most the
+latest one second of visual history. `G` already contains global audio/video
+context, so this tests whether an explicit recent encoder readout adds timely
+evidence. It is not an audio-free Omni baseline.
+
+F2/F3 are no-box full-frame visual-context models. Improvement does not by
+itself establish speaker grounding, identity or addressee reasoning.
+
+## Causal contract
+
+`extract_fullframe_qwen_av_features.py` creates
+`qwen-fullframe-av-tokens-v1` sidecars:
+
+- a cutoff `t` sees only fixed-grid frames with timestamp strictly less than
+  `t`; a frame exactly at `t` is excluded;
+- images are complete EasyCom camera frames after the standard session-banner
+  crop;
+- Qwen post-merger spatial tokens are stored without mean pooling;
+- F3 links only the same merged-grid row/column within one source clip, and
+  resets on clip boundaries, grid changes or excessive time gaps;
+- audio is copied from the existing audited `[t-1,t)` Qwen mixed-audio
+  sidecar; its person-crop tensors are never consumed;
+- visual/audio model identity, config digest and Transformers version must
+  match exactly;
+- partial smoke sidecars are rejected by training.
+
+The targets remain the unchanged 5 Hz `background/wearer/others` labels and
+the ten 0.2--2.0-second horizons.
+
+## 1. Four-second GPU smoke extraction
+
+```bash
+cd /data1/yfzx/wangx/Code/icassp2027-experiment
+
+DATA_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom
+MODEL=/data1/yfzx/wangx/Models/Qwen2.5-Omni-3B
+FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
+TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
+AUDIO_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/local_qwen_1s_encoded_tf4576
+SMOKE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/fullframe_qwen_av_smoke
+
+CUDA_VISIBLE_DEVICES=0 python extract_fullframe_qwen_av_features.py \
+  --data-root "$DATA_ROOT" --feature-root "$FEATURE_ROOT" \
+  --target-root "$TARGET_ROOT" --audio-feature-root "$AUDIO_ROOT" \
+  --output-root "$SMOKE_ROOT" --samples 4 --max-steps 20 \
+  --model-id "$MODEL" --device cuda:0 --dtype bfloat16 \
+  --flash-attention-2 --video-fps 2 --max-pixels 50176 \
+  --image-batch-size 8
+```
+
+Expected: 20 states spanning 0.2--4.0 s; eight visual frames at
+`0.0,...,3.5`; finite nonempty spatial tokens; matched 2048D audio; and
+`partial=True`. Never train on this smoke file.
+
+If the audio sidecars were made with an explicit `--encoder-id`, pass that
+exact value here. Otherwise omit it, as above, so both branches use the same
+recorded local checkpoint path.
+
+Run the regression tests in the remote environment:
+
+```bash
+python -m unittest -v test_fullframe_qwen_av test_local_turn_features
+```
+
+They verify strict boundary exclusion, same-grid F3 history, malformed and
+partial-contract rejection, and F2/F3 forward/backward behavior.
+
+## 2. Extract all sessions
+
+Use a fresh root. Existing files are not overwritten unless `--overwrite` is
+explicitly provided.
+
+```bash
+FULLFRAME_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/fullframe_qwen_av_2fps_224
+
+CUDA_VISIBLE_DEVICES=0 python extract_fullframe_qwen_av_features.py \
+  --data-root "$DATA_ROOT" --feature-root "$FEATURE_ROOT" \
+  --target-root "$TARGET_ROOT" --audio-feature-root "$AUDIO_ROOT" \
+  --output-root "$FULLFRAME_ROOT" \
+  --samples 1 2 3 4 5 6 7 8 9 10 11 12 \
+  --model-id "$MODEL" --device cuda:0 --dtype bfloat16 \
+  --flash-attention-2 --video-fps 2 --max-pixels 50176 \
+  --image-batch-size 8
+```
+
+On interruption, rerun only missing session numbers. Do not mix checkpoints,
+Transformers versions, FPS settings or pixel budgets in one root.
+
+## 3. Train and evaluate
+
+The launcher uses Sessions 4--12 for four fixed epochs and evaluates Sessions
+3/2/1 with the existing LSTR-MCE ten-horizon protocol.
+
+```bash
+export FEATURE_ROOT TARGET_ROOT FULLFRAME_ROOT
+export CHECKPOINT_ROOT=checkpoints/easycom_fullframe_qwen_av
+export RESULT_ROOT=results/easycom_fullframe_qwen_av
+
+CUDA_VISIBLE_DEVICES=0 bash run_easycom_fullframe_qwen_av.sh f2 0
+CUDA_VISIBLE_DEVICES=1 bash run_easycom_fullframe_qwen_av.sh f3 0
+```
+
+After seed 0 passes shape/loss/inference checks:
+
+```bash
+CUDA_VISIBLE_DEVICES=0 bash run_easycom_fullframe_qwen_av.sh f2 1 2
+CUDA_VISIBLE_DEVICES=1 bash run_easycom_fullframe_qwen_av.sh f3 1 2
+```
+
+Defaults are train batch 64, inference batch 128, F3 history 2 frames and
+maximum inter-frame link gap 0.75 s. Lower `TRAIN_BATCH_SIZE` or
+`INFER_BATCH_SIZE` if needed; do not change the feature contract.
+
+The checkpoint-compatible option remains named `--person-history-frames`; in
+F3 it means observations per spatial grid position, not person history.
+
+## Required comparison and claim boundary
+
+Compare matched seeds for `G+A`, F2 and F3. Add a past time-shifted full-frame
+control before attributing gains to AV synchronization. F3 contains a GRU that
+F2 does not, so F3–F2 combines extra history and temporal capacity; a
+parameter-matched audio/query temporal control is required before claiming a
+visual-motion mechanism.
+
+Dense AP/mAP is an architecture screen. Floor-entry/yield interpretations
+remain timing-derived proxies until the competing-risk real-data audit is
+frozen.