Add causal person fusion and MuVAP stage-3 baselines
This commit is contained in:
Xin Wang
Vendored
+233
@@ -0,0 +1,233 @@
|
||||
# MuVAP
|
||||
|
||||
**MuVAP: Multimodal Multiparty Voice Activity Projection for Turn-taking
|
||||
Prediction in the Wild** — Interspeech 2026.
|
||||
|
||||
Demo: https://interspeech2026anonymous-eng.github.io/demo/
|
||||
|
||||
MuVAP is built from modules that share one codebase and are released one at a
|
||||
time. Each is self-contained: it trains, evaluates, and publishes weights on its
|
||||
own, without the modules that have not landed yet.
|
||||
|
||||
| Module | Status | Entry point |
|
||||
| --- | --- | --- |
|
||||
| VAP — voice activity projection from audio | released | `train_vap.py` |
|
||||
| ASD — audio-visual active speaker detection | released | `train_asd.py` |
|
||||
| MuVAP — multiparty fusion of the two | to follow | — |
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
python -m pip install -e . # add ".[mimi]" for the Mimi frontend
|
||||
```
|
||||
|
||||
Install the PyTorch build matching your CUDA version first.
|
||||
|
||||
The frozen audio frontends download their pretrained weights the first time a
|
||||
model is built. Fetch them ahead of time — worth doing before a batch job, on a
|
||||
machine whose compute nodes have no outbound network:
|
||||
|
||||
```bash
|
||||
python tools/fetch_frontends.py # or: fetch_frontends.py cpc
|
||||
```
|
||||
|
||||
|
||||
## VAP
|
||||
|
||||
Voice Activity Projection: a causal model that predicts who will hold the floor
|
||||
next, from audio alone. Every configuration shares one training program and
|
||||
differs only in the config file, so a result difference between them comes from
|
||||
the setup rather than from a different pipeline.
|
||||
|
||||
| Configuration | Config | Input | Codebook |
|
||||
| --- | --- | --- | --- |
|
||||
| Role-based VAP | `config/yaml/vap_role.yaml` | 1 mixed channel | 136 role-relative classes |
|
||||
| Speaker-based VAP | `config/yaml/vap_speaker.yaml` | 2 channels, one per speaker | 256 classes, one channel per row |
|
||||
| Speaker-based, mono | `config/yaml/vap_speaker_mono_mimi.yaml` | 1 mixed channel + VAD | 256 classes, one channel per row |
|
||||
|
||||
The mono setup is the original VAP arrangement: one channel carries the audio
|
||||
and the causal voice activity behind each frame supplies the speaker identity a
|
||||
second channel would have carried. `config/yaml/` also holds the frontend and
|
||||
frame-rate variants of each - `*_mimi_12hz`, `*_cpc_50hz`.
|
||||
|
||||
```bash
|
||||
python train_vap.py --config config/yaml/vap_role.yaml --name role
|
||||
```
|
||||
|
||||
### Positional encoding
|
||||
|
||||
`vap.temporal.pos_encoding` selects `alibi`, causal `rope`, `sinusoidal`, or
|
||||
`none`. The stack stays causal regardless — only how position reaches attention
|
||||
changes.
|
||||
|
||||
See [`docs/vap.md`](docs/vap.md) for what each setup means, what it logs, how to
|
||||
prepare Fisher, and the exact causality and alignment contracts.
|
||||
|
||||
|
||||
## ASD
|
||||
|
||||
Audio-visual active speaker detection: given a face track and the audio around
|
||||
it, decide frame by frame whether that face is speaking. The reported number is
|
||||
the official AVA-ActiveSpeaker mAP.
|
||||
|
||||
| Configuration | Config | Frontend |
|
||||
| --- | --- | --- |
|
||||
| ASD | `config/yaml/asd.yaml` | CPC |
|
||||
| ASD + Mimi | `config/yaml/asd_mimi.yaml` | Mimi |
|
||||
|
||||
The frontends are the same two the VAP module uses, and both emit features at
|
||||
the 25 Hz rate the visual frontend runs at, so they substitute for each other
|
||||
here too.
|
||||
|
||||
```bash
|
||||
python train_asd.py --config config/yaml/asd.yaml --name asd
|
||||
```
|
||||
|
||||
Faces reach the model as 112x112 grayscale, the geometry the packs are written
|
||||
in - see [preprocessing](docs/preprocessing.md) for where that is fixed and why.
|
||||
On top of it, `asd.augment` applies one geometric transform per chunk, drawn
|
||||
uniformly from none, horizontal flip, random crop rescaled back to 112, and
|
||||
rotation; `crop_scale` is the side of that crop as a fraction of the frame.
|
||||
Audio is augmented separately: with probability `audio_mix_prob` a donor clip
|
||||
is mixed in at an SNR drawn from `audio_snr_db` while the label stays unchanged,
|
||||
so the model has to reject speech it cannot see on the face.
|
||||
|
||||
```yaml
|
||||
augment:
|
||||
audio_mix_prob: 0.5 # chance of mixing in off-screen speech
|
||||
audio_snr_db: [-5.0, 5.0] # how loud that donor is against the real audio
|
||||
visual: true # geometric transforms on the face track
|
||||
crop_scale: [0.7, 1.0] # random-crop side, as a fraction of 112
|
||||
rotate_degrees: 15.0 # maximum rotation
|
||||
substitute_audio_prob: 0.0 # ablation: replace the audio, relabel silent
|
||||
```
|
||||
|
||||
Every source corpus is packed into one memory-mapped format first. A pack is a
|
||||
decoded mirror of the corpus, not a model-ready tensor set: grayscale face
|
||||
crops and one label per frame at the *native* video rate, plus 16 kHz audio at
|
||||
its original gain. Sampling onto the 25 Hz model timeline and loudness
|
||||
normalization happen in the dataloader, so one pack can serve models with
|
||||
different conventions.
|
||||
|
||||
```bash
|
||||
python -m preprocess.asd.prepare ava --split train \
|
||||
--loader /data/AVA/csv/train_loader.csv \
|
||||
--visual-root /data/AVA/clips_videos --audio-root /data/AVA/clips_audios
|
||||
python -m preprocess.asd.validate /data/AVA/packed/train
|
||||
```
|
||||
|
||||
`python -m preprocess.asd.prepare --help` lists the accepted source schemas.
|
||||
See [`docs/preprocessing.md`](docs/preprocessing.md) and
|
||||
[`docs/data_alignment.md`](docs/data_alignment.md) for the storage and
|
||||
alignment contracts.
|
||||
|
||||
**Audio convention.** The model expects audio normalized to -25 dBFS RMS,
|
||||
capped at 20 dB of gain and a 0.99 peak, with the level measured over the whole
|
||||
recording rather than the evaluated slice. This is not the raw-waveform
|
||||
convention other ASD codebases use, and a checkpoint evaluated on unnormalized
|
||||
audio will not reproduce the mAP it was trained to. Packs store audio
|
||||
unnormalized and the loader applies this; the constants live in
|
||||
`data/media.py`.
|
||||
|
||||
|
||||
## Running either module
|
||||
|
||||
`--set KEY=VALUE` overrides any config value (dotted path, YAML-parsed) and can
|
||||
be repeated; it fails on a key the config does not already define, so a typo
|
||||
cannot silently do nothing. `--wandb` enables tracking, `--name` chooses the
|
||||
run directory.
|
||||
|
||||
`checkpoint_monitor` picks what `best.ckpt` selects on — `val/loss` for VAP,
|
||||
`val/mAP_official` for ASD. Clear it where there is no held-out split to select
|
||||
on, and the final epoch is the only artifact the run keeps:
|
||||
|
||||
```bash
|
||||
python train_vap.py --config config/yaml/vap_role.yaml --set vap.checkpoint_monitor=
|
||||
```
|
||||
|
||||
`--test` then falls back to `last` instead of `best`. See
|
||||
[Weight releases](#weight-releases) for what each checkpoint is.
|
||||
|
||||
## Audio frontends
|
||||
|
||||
`vap.audio_encoder` selects the frozen frontend. Both emit features at the rate
|
||||
the projection window labels, so they are drop-in alternatives at a shared rate.
|
||||
|
||||
| | `cpc` (default) | `mimi` |
|
||||
| --- | --- | --- |
|
||||
| Weights | CPC, LibriLight 60k | `kyutai/mimi`, from Moshi |
|
||||
| Frozen params | 1.8M | 37.8M |
|
||||
| Feature dim | 256 | 512 |
|
||||
| Frame rates | 100 / 50 / 25 / 12.5 | 25 / 12.5 |
|
||||
|
||||
CPC halves a 100 Hz convolutional stack down to the requested rate; Mimi taps
|
||||
either its 25 Hz encoder output or its 12.5 Hz latent, so 50 Hz is a CPC-only
|
||||
option.
|
||||
|
||||
Both are causal, which is what makes the model usable as a streaming predictor.
|
||||
|
||||
|
||||
## Weight releases
|
||||
|
||||
Trained weights live on the Hub at
|
||||
[Haotian-Qi/MuVAP](https://huggingface.co/Haotian-Qi/MuVAP).
|
||||
|
||||
Two ASD releases, `asd-cpc` and `asd-mimi`, scoring 90.50 and 92.04
|
||||
`mAP_official` on AVA-ActiveSpeaker. The VAP releases span three codebooks, two
|
||||
frontends and three frame rates - [docs/vap.md](docs/vap.md#released-models)
|
||||
lists them with their scores.
|
||||
|
||||
Fetch one, then evaluate it:
|
||||
|
||||
```bash
|
||||
huggingface-cli download Haotian-Qi/MuVAP --include 'vap-role-mimi/*' --local-dir weights
|
||||
|
||||
python train_vap.py --config config/yaml/vap_role.yaml \
|
||||
--test --weights weights/vap-role-mimi \
|
||||
--set fisher_path=/path/to/fisher
|
||||
```
|
||||
|
||||
That reproduces the score in the table exactly. All of them at once:
|
||||
|
||||
```bash
|
||||
huggingface-cli download Haotian-Qi/MuVAP --local-dir weights
|
||||
```
|
||||
|
||||
A run produces two checkpoints. `last.ckpt` is the final epoch — the artifact
|
||||
the run committed to. `best.ckpt` is the epoch a validation metric picked, and
|
||||
exists only when `checkpoint_monitor` is set; where there is no held-out split
|
||||
to select on, clear it and the final epoch is the only artifact.
|
||||
|
||||
Publish either one with:
|
||||
|
||||
```bash
|
||||
python tools/export_checkpoint.py <run>/last.ckpt <release dir> --module vap
|
||||
```
|
||||
|
||||
A release carries the trained weights alone: no optimizer state, and no copy of
|
||||
the frozen audio frontend, which is fetched from its own pretrained source when
|
||||
the model is built. It ships `config.yaml` (the architecture) and
|
||||
`provenance.json`, which records the checkpoint it came from and what that
|
||||
checkpoint scored. Point `--weights` at a release directory to evaluate it:
|
||||
|
||||
```bash
|
||||
python train_vap.py --config config/yaml/vap_role.yaml --test --weights <release>
|
||||
```
|
||||
|
||||
The architecture comes from the release's own `config.yaml`; dataset roots and
|
||||
output directories come from the `--config` you pass, so a release runs anywhere
|
||||
without reproducing the paths it was trained with. A release that does not fit
|
||||
the config it is given fails immediately, naming the offending weights.
|
||||
|
||||
Or load one directly:
|
||||
|
||||
```python
|
||||
import yaml
|
||||
from models.release import load_weights, resolve
|
||||
from models.vap import build_vap
|
||||
|
||||
weights, config = resolve("path/to/release")
|
||||
cfg = yaml.safe_load(open(config))["vap"]
|
||||
model = load_weights(build_vap(cfg), weights).eval()
|
||||
logits = model(waveform) # [batch, frames, classes] at the release's rate
|
||||
```
|
||||
Reference in New Issue
Block a user