wx44wx/icassp2027-experiment
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add causal person fusion and MuVAP stage-3 baselines

Browse Source
This commit is contained in:
Xin Wang
2026-09-10 06:54:22 +08:00
parent 0534c9d3d0
commit d426b4740a
225 changed files with 27716 additions and 119 deletions
Download Patch File Download Diff File Expand all files Collapse all files
third_party/MuVAP/docs/vap.md
Vendored
+232
View File
@@ -0,0 +1,232 @@
# The VAP module
Voice Activity Projection setups share one training program. They differ in
config only - how many audio channels the model reads, which codebook
`ProjectionWindow` projects the VAD onto, and which frozen audio frontend runs
in front - so any result difference between them comes from the setup rather
than from a different pipeline.
| Experiment | Config | Frontend | Channels | `mode` | Classes |
| --- | --- | --- | --- | --- | --- |
| Speaker-based VAP | `vap_speaker.yaml` | CPC | 2 (one per speaker) | `speaker_based` | 256 |
| Role-based VAP | `vap_role.yaml` | CPC | 1 (downmix) | `role_relative` | 136 |
Hold and shift come from the logits in both setups: `role_relative` names the
roles in the class, and `speaker_based` names the channels, which needs
`prev_spk` to say which one held the floor.
## What each setup is
**Role-Based** is the MuVAP+ default. Both speakers are mixed into one channel,
so the model is never told who is who. The class names the *roles* - current
speaker and next speaker - through an order-invariant pair codebook over two
history and two future bins. Hold and shift fall straight out of the logits
because the class already says which role continues.
**Speaker-based VAP** reproduces Ekstedt & Skantze (2022): one audio channel per
speaker, a shared encoder and causal self-attention stack applied to each
channel, cross-attention between the two channels, and one head over their
concatenation. The projection window works per channel - one channel per row of
the code - giving `2 speakers x 4 future bins = 8 bits = 256` classes. Because
the classes name *channels* rather than roles, reading hold and shift needs to
know which channel held the floor, which is why the event manifest's
`prev_spk` column is carried into the test step.
## Audio frontend
`vap.audio_encoder` selects the frozen frontend. Both land on the same 25 Hz
grid the projection window labels, so they are drop-in alternatives:
| | `cpc` (default) | `mimi` |
| --- | --- | --- |
| Weights | CPC, LibriLight 60k | `kyutai/mimi`, from Moshi |
| Frozen params | 1.8M | 37.8M |
| Feature dim | 256 | 512 |
| Native rate | 16 kHz | 24 kHz (resampled causally inside the encoder) |
```yaml
audio_encoder:
name: mimi
tap: encoder # 25 Hz. `latent` is Mimi's own 12.5 Hz rate
```
Mimi is causal by construction - its SEANet convolutions left-pad and its
encoder transformer uses a sliding causal mask. Getting there causally
needs care in one place: `torchaudio`'s resampler convolves with a symmetric
sinc, so a plain 16 -> 24 kHz resample leaks about 9 output samples of future.
`causal_resample_16k_to_24k` delays the signal by 1 ms instead, which removes
the leak at the cost of a 1 ms lag against a 40 ms frame.
The `latent` tap is refused at build time under a 25 Hz codebook rather than
failing later on a shape mismatch.
Mimi's 512-dim features are bridged onto `d_model` by the model's encoder
projection, so `d_model` need not change.
## Positional encoding
`vap.temporal.pos_encoding` selects how position reaches attention. It never
changes what a frame can see: every temporal stack is causal, frame `t` attends
to frames `<= t` only, so a trained model stays usable as a streaming predictor.
- `alibi` - per-head linear distance penalty on the logits (the previous default).
- `rope` - **causal rotary position embedding**. Query and key are rotated by an
angle proportional to the absolute frame index, so the attention logit depends
only on the distance between two frames. Under the causal mask that distance is
never negative, and no future frame can leak in through the encoding. A query
shorter than its key stream is offset to the end of that stream, so streaming
inference with a cached history lines up with training.
- `sinusoidal` - an additive input-side embedding, for ablation.
- `none` - no positional information beyond the causal mask itself.
`ffn` (`mlp` / `swiglu`) and `norm` (`layer` / `rms`) are independent knobs on
the same stack.
Switch encoding without editing the file:
```bash
python train_vap.py --config config/yaml/vap_role.yaml --set vap.temporal.pos_encoding=rope
```
## Data
Two paths in each config are the only machine-specific values:
```yaml
fisher_path: data/fisher # the corpus
checkpoint_dir: runs/vap # checkpoints and logs
```
`checkpoint_dir` is also the trainer's root, so Lightning's logs land beside
the checkpoints.
The corpus tree looks like this:
```text
fisher/
p1/ p2/
audio/<group>/<conv>.wav # source, stereo
regions_vap/<group>/<conv>_*_words.txt
seg/audio/<group>/<conv>/<i>.npy # 30 s stereo segment
seg/vad/<group>/<conv>/<i>.npy # 2 x (750 + 2*context) VAD frames
tune/audio/... # 10 s mono event clips
tune/audio_stereo/... # 10 s stereo event clips (original VAP only)
train.txt val.txt test.txt
turn_events.txt
train_events.txt val_events.txt test_events.txt
```
**Labels are not stored.** Segments hold the raw two-speaker VAD with
`--context-sec` of extra frames on each side - everything the widest projection
window needs - and the dataloader projects them at load time. Changing
`projection_window.mode` between the three setups is therefore a config edit,
not a re-run of preprocessing.
Build the tree:
```bash
python preprocess/vap/00_prep_fisher.py --config config/yaml/vap_role.yaml # segments + VAD
python preprocess/vap/01_create_event.py --config config/yaml/vap_role.yaml # turn events
python preprocess/vap/02_prep_event.py --config config/yaml/vap_role.yaml # mono event clips
```
The speaker-based setup evaluates on two-channel clips. `config/yaml/vap_speaker.yaml`
ships with `source: raw`, which reads them straight from the source recording,
so nothing more is needed. To evaluate it from pre-cut clips instead, write a
two-channel copy - it lands in `tune/audio_stereo`, leaving the mono clips
untouched - and set `source: npy`:
```bash
python preprocess/vap/02_prep_event.py --config config/yaml/vap_role.yaml --channels 2
```
`--splits` defaults to `val test`, which is what evaluation reads. Adding
`train` cuts clips for roughly 2.5 M more events and is only worth it to
enlarge the probe's tuning pool.
If a projection window needs more context than a segment stores, the dataloader
says so by name and asks for a wider `--context-sec`; the 2 s default covers all
three setups shipped here.
## Running
```bash
# Role-Based, the default
python train_vap.py --config config/yaml/vap_role.yaml --name role_alibi
# Role-Based with causal RoPE
python train_vap.py --config config/yaml/vap_role.yaml --name role_rope \
--set vap.temporal.pos_encoding=rope
# Speaker-based VAP, two channels
python train_vap.py --config config/yaml/vap_speaker.yaml --name speaker
```
Evaluate a checkpoint without training:
```bash
python train_vap.py --config config/yaml/vap_role.yaml --test --checkpoint runs/vap_role/role_alibi/best.ckpt
```
Add `--wandb` for tracking, `--seed` to change the seed, and `--name` to choose
the checkpoint directory. `--set KEY=VALUE` overrides any existing config key
(dotted path, value parsed as YAML) and can be repeated; it fails loudly on a
key the config does not already define, so a typo cannot silently do nothing.
## Released models
Weights are on the Hub at [Haotian-Qi/MuVAP](https://huggingface.co/Haotian-Qi/MuVAP).
`f1_macro` is zero-shot hold/shift on the Fisher events, decoded from the logits
over `p_future`, at the shift prior each codebook defaults to.
| Release | Codebook | Audio in | Rate | Frontend | f1_macro | bacc | probe f1 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| `vap-speaker-mimi-12hz` | speaker, 256 | 2 ch | 12.5 | Mimi | **0.7776** | 0.7495 | 0.7714 |
| `vap-speaker-mimi` | speaker, 256 | 2 ch | 25 | Mimi | 0.7755 | 0.7464 | 0.7716 |
| `vap-role-mimi` | role, 136 | 1 ch | 25 | Mimi | 0.7589 | 0.7698 | 0.7623 |
| `vap-role-mimi-12hz` | role, 136 | 1 ch | 12.5 | Mimi | 0.7565 | 0.7663 | 0.7570 |
| `vap-speaker-mono-mimi` | speaker, 256 | 1 ch + VAD | 25 | Mimi | 0.7517 | 0.7216 | 0.7619 |
| `vap-speaker-mono-mimi-12hz` | speaker, 256 | 1 ch + VAD | 12.5 | Mimi | 0.7484 | 0.7183 | 0.7585 |
| `vap-speaker-cpc-50hz` | speaker, 256 | 2 ch | 50 | CPC | 0.7444 | 0.7129 | 0.7520 |
| `vap-role-cpc-50hz` | role, 136 | 1 ch | 50 | CPC | 0.7358 | 0.7378 | 0.7382 |
| `vap-speaker-cpc` | speaker, 256 | 2 ch | 25 | CPC | 0.7310 | 0.6983 | 0.7508 |
| `vap-role-cpc` | role, 136 | 1 ch | 25 | CPC | 0.7289 | 0.7303 | 0.7304 |
| `vap-speaker-mono-cpc-50hz` | speaker, 256 | 1 ch + VAD | 50 | CPC | 0.7148 | 0.6823 | 0.7398 |
Read it with three cautions.
**The prior is a threshold, and it differs by codebook.** Scaling the shift slot
by `s` and renormalising is exactly a decision threshold at `1/(1+s)`, so the
speaker rows sit at 0.5 and the role rows at 0.333. The `bacc` column is where
that shows. Comparisons within a codebook are sound; across codebooks they are
comparing operating points as much as models. `probe f1`, a logistic probe on
the last-frame embedding, is threshold-free and closer to like-for-like.
**The frontend is the only large effect.** Mimi over CPC is +0.045 on speaker and
+0.030 on role at matched rate. Frame rate is 3-6x smaller: 25 -> 50 Hz on CPC
gains 0.013 (speaker) and 0.007 (role), and on Mimi 25 -> 12.5 Hz costs nothing
measurable - +0.002, -0.002, -0.003 across the three codebooks. Matching a
12.5 Hz downstream system is therefore close to free.
**The `-mono-` releases need VAD at inference.** They take one mixed channel plus
the voice activity behind each frame, `model(audio, vad)`, which is what supplies
the speaker identity a second channel would carry. Quality depends on the VAD you
feed them.
## Metrics
`trainer.test` runs two dataloaders: the tune events fit the linear probe, the
test events are scored. Events are pooled three ways - `all`, `SILENT`,
`ACTIVE` - and three prefixes keep the panels apart:
- `test/<pool>/f1_macro`, `test/<pool>/bacc` - the headline. Hold and shift are
decoded from the logits, zero shot, over `p_future` at the shift prior the
codebook defaults to: 2.0 for `role_relative`, 1.0 for `speaker_based`, whose
rows name channels and so also need the event's `prev_spk`.
- `probe/<pool>/f1_macro`, `probe/<pool>/bacc` - a logistic probe on the
last-frame embedding. It bypasses the codebook, so the gap against `test/`
is what the readout costs rather than anything about the model. Being fitted,
it is a diagnostic and not a score.
- `ablation/<pool>/f1_macro_scale_<s>` - the shift prior swept 1.0 ... 3.0, the
way the VAP papers report it. `speaker_based` gains about 0.045 from a prior
of 2.0; the role setups are already at their optimum and gain 0.001.