Global Omni + shared 25 Hz audio + per-person evidence

What this implementation does

The first version deliberately keeps the two jobs separate:

Existing Qwen2.5-Omni states provide the global scene/conversation context G_t at the verified 5 Hz task grid.
MuVAP's frozen CPC frontend encodes the dense session's mixed 16 kHz audio once into a shared causal 25 Hz stream U_t; it does not reset at the artificial one-minute file boundaries.
MuVAP's causal visual track encoder produces one visual state per visible person every 40 ms. Its audio-visual active-speaker readout contributes one speaking logit, so a person token is [visual_state, ASD_score]; the shared audio vector is not duplicated inside every person token.
At training time G_t + U_t queries the variable-sized person set using masked attention.
A zero-initialized residual readout maps the selected person evidence into a correction of the ordinary G_t + U_t logits. With matching shared weights, the initial model is exactly the audio-only baseline; the person branch must earn every change during training.
The ten-horizon floor head predicts the same 0.2--2.0 second EgoSpeak target.

In compact form:

full scene + mixed audio -> frozen Omni -> G_t --------+--> G+U base logits -----+
mixed audio -> frozen CPC once -> U_t -----------------+                        sum
                                                                                ^
25 Hz face track -> causal visual state --+                                     |
U_t + visual state -> ASD speaking score -+-> {P_t,i} -> zero-init residual ----+
                                      G_t+U_t queries person set

The person branch uses MuVAP's released ASD module, not the unreleased/full MuVAP multiparty forecasting module. This is a traditional high-rate active-speaker frontend combined with an Omni global representation.

Causality and rate contract

MuVAP ASD runs at 25 Hz. Model state k consumes the complete interval [k/25, (k+1)/25) and is timestamped by its right edge (k+1)/25.
Every fifth ASD state ends on a 5 Hz task cutoff. Equality is causal here: the saved state at 0.2s has consumed samples strictly before 0.2s.
MuVAP-style chunks score 10 seconds at a time and prepend at most 2 seconds of past context. Chunks overlap only into the past.
A source-clip boundary or missing face observation resets a local face track. The shared audio stream remains continuous on the same concatenated dense timeline used by the targets and global Omni features.
The released MuVAP data loader's whole-recording RMS normalization is not reused: at an early cutoff that statistic would depend on future audio. The extractor uses fixed float-PCM scaling with pointwise clipping only. Record this causal deployment/domain-shift choice when reporting the experiment.
Speech labels, wearer targets and future frames are never extractor inputs.
Provided boxes isolate the representation/fusion question. They are not an online detector result and must be described that way in the paper.

External dependency

The audited MuVAP checkout and published ASD-CPC release are vendored under third_party/MuVAP, including an immutable .upstream-revision record. For a fresh checkout of this project, no separate MuVAP clone or weight download is needed. The following commands document how that vendored snapshot was obtained and are only needed when intentionally refreshing it:

git clone https://github.com/Haotian-Qi/MuVAP.git third_party/MuVAP
git -C third_party/MuVAP checkout a1ef9c8
python third_party/MuVAP/tools/fetch_frontends.py cpc

huggingface-cli download Haotian-Qi/MuVAP \
  --include 'asd-cpc/*' \
  --local-dir third_party/MuVAP/weights

The extractor imports the pinned vendored checkout directly; it does not need an editable MuVAP install. Check that torch, torchaudio, einops and yaml import in the existing Qwen container instead of upgrading its full dependency stack.

The extractor checks the source revision and hashes the selected release weights into every output contract. If a newer MuVAP revision is intentionally tested, give its exact commit with --muvap-revision and use a new output root.

EasyCom smoke extraction

Run a 10-second partial smoke first. A partial sidecar is intentionally rejected by training.

export DATASET_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom
export FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
export HIGHRATE_LOCAL_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/local_muvap_asd_25hz_v2
export ASD_WEIGHTS=third_party/MuVAP/weights/asd-cpc

CUDA_VISIBLE_DEVICES=0 python extract_highrate_person_av_features.py \
  --dataset easycom \
  --data-root "$DATASET_ROOT" \
  --feature-root "$FEATURE_ROOT" \
  --output-root features/highrate_person_av_smoke \
  --samples 12 \
  --muvap-root third_party/MuVAP \
  --asd-weights "$ASD_WEIGHTS" \
  --max-steps 50 \
  --device cuda:0 \
  --dtype bfloat16

Audit the result:

python - <<'PY'
import torch

p = torch.load("features/highrate_person_av_smoke/Session_12.pt", map_location="cpu", weights_only=False)
print("format:", p["format"])
print("partial:", p["partial"])
print("times:", p["times"][:3].tolist(), "...", p["times"][-1].item())
print("person features:", tuple(p["persons"]["features"].shape))
print("shared audio:", tuple(p["audio"].shape))
print("frame offsets:", tuple(p["persons"]["frame_offsets"].shape))
print("person-row coverage:", float(p["visual_valid"].float().mean()))
print("uses speech labels:", p["config"]["uses_speech_labels"])
print("causal policy:", p["config"]["causal_policy"])
PY

Expected invariants are causal-highrate-person-av-v2, partial=True, times [0.2, 0.4, ...], shared audio width 256, person width 257 (256D causal visual state + 1 AV activity logit), and uses_speech_labels=False.

Full EasyCom extraction

Run one command per GPU/terminal. The shard assignments are disjoint and the destination contains real files, not symbolic links:

CUDA_VISIBLE_DEVICES=0 python extract_highrate_person_av_features.py \
  --dataset easycom --data-root "$DATASET_ROOT" --feature-root "$FEATURE_ROOT" \
  --output-root "$HIGHRATE_LOCAL_ROOT" --samples 1 2 3 4 5 6 7 8 9 10 11 12 \
  --num-shards 8 --shard-index 0 --muvap-root third_party/MuVAP \
  --asd-weights "$ASD_WEIGHTS" --device cuda:0 --dtype bfloat16

Repeat with physical GPU and shard index 1 through 7. Do not add --overwrite unless the exact file has been inspected and a replacement is intentional.

First matched training screen

The screen trains on Sessions 4--11 and uses Session 12 for checkpoint selection. It does not inspect Sessions 1--3. Run different seeds on different GPUs:

CUDA_VISIBLE_DEVICES=0 bash run_easycom_highrate_person_av.sh all 0
CUDA_VISIBLE_DEVICES=1 bash run_easycom_highrate_person_av.sh all 1
CUDA_VISIBLE_DEVICES=2 bash run_easycom_highrate_person_av.sh all 2

This compares four matched conditions:

global: frozen Omni global state G only;
qwen_audio: existing G+A_qwen strong baseline;
traditional_audio: global Omni G plus one shared CPC stream U;
highrate: load that seed's trained G+U checkpoint, freeze the exact base, and learn only the attention/residual over frozen 25 Hz causal ASD person states. This first screen therefore asks whether the new person evidence adds information, without allowing the base to drift.

The all command runs traditional_audio before highrate, so the initializer exists. If running separately, run traditional_audio first for the same seed.

Only after selecting the architecture on Session 12 should the fixed checkpoints be run on the official test set:

CUDA_VISIBLE_DEVICES=0 bash run_easycom_highrate_person_av.sh test 0
CUDA_VISIBLE_DEVICES=1 bash run_easycom_highrate_person_av.sh test 1
CUDA_VISIBLE_DEVICES=2 bash run_easycom_highrate_person_av.sh test 2

Dense AP/mAP is a screening/comparability result. The event evaluator still needs competing-risk censoring before claims about false entries, responses or yields are final.

8.1 KiB Raw Blame History