icassp2027-experiment/docs/easycom_highrate_person_av.md

# Global Omni + shared 25 Hz audio + per-person evidence

## What this implementation does

The first version deliberately keeps the two jobs separate:

1. Existing Qwen2.5-Omni states provide the global scene/conversation context
   `G_t` at the verified 5 Hz task grid.
2. MuVAP's frozen CPC frontend encodes the dense session's mixed 16 kHz audio
   once into a shared causal 25 Hz stream `U_t`; it does not reset at the
   artificial one-minute file boundaries.
3. MuVAP's causal visual track encoder produces one visual state per visible
   person every 40 ms. Its audio-visual active-speaker readout contributes one
   speaking logit, so a person token is `[visual_state, ASD_score]`; the shared
   audio vector is not duplicated inside every person token.
4. At training time `G_t + U_t` queries the variable-sized person set using
   masked attention.
5. A zero-initialized residual readout maps the selected person evidence into
   a correction of the ordinary `G_t + U_t` logits.  With matching shared
   weights, the initial model is exactly the audio-only baseline; the person
   branch must earn every change during training.
6. The ten-horizon floor head predicts the same 0.2--2.0 second EgoSpeak target.

In compact form:

```text
full scene + mixed audio -> frozen Omni -> G_t --------+--> G+U base logits -----+
mixed audio -> frozen CPC once -> U_t -----------------+                        sum
                                                                                ^
25 Hz face track -> causal visual state --+                                     |
U_t + visual state -> ASD speaking score -+-> {P_t,i} -> zero-init residual ----+
                                      G_t+U_t queries person set
```

The person branch uses MuVAP's released **ASD module**, not the unreleased/full
MuVAP multiparty forecasting module.  This is a traditional high-rate
active-speaker frontend combined with an Omni global representation.

## Causality and rate contract

- MuVAP ASD runs at 25 Hz.  Model state `k` consumes the complete interval
  `[k/25, (k+1)/25)` and is timestamped by its right edge `(k+1)/25`.
- Every fifth ASD state ends on a 5 Hz task cutoff.  Equality is causal here:
  the saved state at `0.2s` has consumed samples strictly before `0.2s`.
- MuVAP-style chunks score 10 seconds at a time and prepend at most 2 seconds
  of past context.  Chunks overlap only into the past.
- A source-clip boundary or missing face observation resets a local face track.
  The shared audio stream remains continuous on the same concatenated dense
  timeline used by the targets and global Omni features.
- The released MuVAP data loader's whole-recording RMS normalization is not
  reused: at an early cutoff that statistic would depend on future audio.  The
  extractor uses fixed float-PCM scaling with pointwise clipping only.  Record
  this causal deployment/domain-shift choice when reporting the experiment.
- Speech labels, wearer targets and future frames are never extractor inputs.
- Provided boxes isolate the representation/fusion question.  They are not an
  online detector result and must be described that way in the paper.

## External dependency

The audited MuVAP checkout and published ASD-CPC release are vendored under
`third_party/MuVAP`, including an immutable `.upstream-revision` record. For a
fresh checkout of this project, no separate MuVAP clone or weight download is
needed. The following commands document how that vendored snapshot was
obtained and are only needed when intentionally refreshing it:

```bash
git clone https://github.com/Haotian-Qi/MuVAP.git third_party/MuVAP
git -C third_party/MuVAP checkout a1ef9c8
python third_party/MuVAP/tools/fetch_frontends.py cpc

huggingface-cli download Haotian-Qi/MuVAP \
  --include 'asd-cpc/*' \
  --local-dir third_party/MuVAP/weights
```

The extractor imports the pinned vendored checkout directly; it does not need
an editable MuVAP install. Check that `torch`, `torchaudio`, `einops` and
`yaml` import in the existing Qwen container instead of upgrading its full
dependency stack.

The extractor checks the source revision and hashes the selected release
weights into every output contract.  If a newer MuVAP revision is intentionally
tested, give its exact commit with `--muvap-revision` and use a new output root.

## EasyCom smoke extraction

Run a 10-second partial smoke first.  A partial sidecar is intentionally
rejected by training.

```bash
export DATASET_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom
export FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
export HIGHRATE_LOCAL_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/local_muvap_asd_25hz_v2
export ASD_WEIGHTS=third_party/MuVAP/weights/asd-cpc

CUDA_VISIBLE_DEVICES=0 python extract_highrate_person_av_features.py \
  --dataset easycom \
  --data-root "$DATASET_ROOT" \
  --feature-root "$FEATURE_ROOT" \
  --output-root features/highrate_person_av_smoke \
  --samples 12 \
  --muvap-root third_party/MuVAP \
  --asd-weights "$ASD_WEIGHTS" \
  --max-steps 50 \
  --device cuda:0 \
  --dtype bfloat16
```

Audit the result:

```bash
python - <<'PY'
import torch

p = torch.load("features/highrate_person_av_smoke/Session_12.pt", map_location="cpu", weights_only=False)
print("format:", p["format"])
print("partial:", p["partial"])
print("times:", p["times"][:3].tolist(), "...", p["times"][-1].item())
print("person features:", tuple(p["persons"]["features"].shape))
print("shared audio:", tuple(p["audio"].shape))
print("frame offsets:", tuple(p["persons"]["frame_offsets"].shape))
print("person-row coverage:", float(p["visual_valid"].float().mean()))
print("uses speech labels:", p["config"]["uses_speech_labels"])
print("causal policy:", p["config"]["causal_policy"])
PY
```

Expected invariants are `causal-highrate-person-av-v2`, `partial=True`, times
`[0.2, 0.4, ...]`, shared audio width 256, person width 257 (`256D causal
visual state + 1 AV activity logit`), and `uses_speech_labels=False`.

## Full EasyCom extraction

Run one command per GPU/terminal.  The shard assignments are disjoint and the
destination contains real files, not symbolic links:

```bash
CUDA_VISIBLE_DEVICES=0 python extract_highrate_person_av_features.py \
  --dataset easycom --data-root "$DATASET_ROOT" --feature-root "$FEATURE_ROOT" \
  --output-root "$HIGHRATE_LOCAL_ROOT" --samples 1 2 3 4 5 6 7 8 9 10 11 12 \
  --num-shards 8 --shard-index 0 --muvap-root third_party/MuVAP \
  --asd-weights "$ASD_WEIGHTS" --device cuda:0 --dtype bfloat16
```

Repeat with physical GPU and shard index `1` through `7`.  Do not add
`--overwrite` unless the exact file has been inspected and a replacement is
intentional.

## First matched training screen

The screen trains on Sessions 4--11 and uses Session 12 for checkpoint
selection.  It does not inspect Sessions 1--3.  Run different seeds on
different GPUs:

```bash
CUDA_VISIBLE_DEVICES=0 bash run_easycom_highrate_person_av.sh all 0
CUDA_VISIBLE_DEVICES=1 bash run_easycom_highrate_person_av.sh all 1
CUDA_VISIBLE_DEVICES=2 bash run_easycom_highrate_person_av.sh all 2
```

This compares four matched conditions:

- `global`: frozen Omni global state `G` only;
- `qwen_audio`: existing `G+A_qwen` strong baseline;
- `traditional_audio`: global Omni `G` plus one shared CPC stream `U`;
- `highrate`: load that seed's trained `G+U` checkpoint, freeze the exact base,
  and learn only the attention/residual over frozen 25 Hz causal ASD person
  states.  This first screen therefore asks whether the new person evidence
  adds information, without allowing the base to drift.

The `all` command runs `traditional_audio` before `highrate`, so the initializer
exists. If running separately, run `traditional_audio` first for the same seed.

Only after selecting the architecture on Session 12 should the fixed
checkpoints be run on the official test set:

```bash
CUDA_VISIBLE_DEVICES=0 bash run_easycom_highrate_person_av.sh test 0
CUDA_VISIBLE_DEVICES=1 bash run_easycom_highrate_person_av.sh test 1
CUDA_VISIBLE_DEVICES=2 bash run_easycom_highrate_person_av.sh test 2
```

Dense AP/mAP is a screening/comparability result.  The event evaluator still
needs competing-risk censoring before claims about false entries, responses or
yields are final.