icassp2027-experiment/third_party/MuVAP/docs/preprocessing.md

# Canonical ASD preprocessing

All datasets are converted to the same versioned `muvap-asd-mmap` format.
Dataset adapters only resolve source metadata; `preprocess.asd.writer` performs
all media conversion and alignment.

## Packed format

A pack is a **decoded mirror of the source corpus, not a model-ready tensor
set**. Faces and labels keep the native frame rate, audio keeps its full
duration and its original gain. Everything MuVAP+ specific - sampling onto the
25 Hz model timeline, loudness normalization, chunking - happens in
`data.dataloaders.packed` when a batch is built. That split is what lets a pack
serve a model whose conventions differ from this one.

Each corpus keeps its pack beside its own media, so `--output` defaults to
`<dataset root>/packed/<split>` and a dataset stays self-contained:

```text
AVA/packed/train/
  dataset.json
  shard-000000/
    faces.npy   # uint8 [native_frames, 112, 112], grayscale
    audio.npy   # int16 mono 16 kHz, unnormalized, full clip duration
    labels.npy  # uint8, one label per NATIVE frame
    index.json  # offsets, source_fps, source_frames, audio level statistics
```

Each shard is approximately 1 GB. The arrays are uncompressed and opened with
`mmap_mode="r"`, so workers and DDP ranks share the operating-system page cache
without HDF5 handles or a per-process copy. Copy complete dataset directories
to node-local storage before training if the shared filesystem is the
bottleneck.

The per-sample invariant is:

```text
source_frames faces == source_frames labels
audio_samples == round(source_frames / source_fps * 16000)
```

### What the reader adds

`index.json` carries `source_fps`, `source_frames`, and the recording's
`audio_mean`, `audio_rms`, and `audio_peak` measured after DC removal. Those
statistics are what let the loader reproduce a whole-recording normalization
from any slice: gain computed from a chunk alone would drift with that chunk's
content. The loader removes DC, scales to -25 dBFS RMS, caps gain at 20 dB and
the peak at 0.99, and samples faces and labels onto the 25 Hz timeline with one
shared index so a label stays attached to the face it describes.

The one convention still baked into the pack is the face crop: 112x112
grayscale, recorded in `dataset.json` as `face_shape` and `face_color`. It is
the ASD community's standard geometry rather than a MuVAP+ choice, but a reader
should check it rather than assume it.

The size is a constant in two places, not a config key: `FACE_SIZE` in
`preprocess/asd/schema.py`, which the writer resizes every JPEG face to, and
`VisualSpec` in `data/media.py`, which is what the loader and the model expect.
Changing the geometry means editing both and re-running preprocessing - a pack
written at one size cannot be read at another.

## AVA

```bash
python -m preprocess.asd.prepare ava \
  --loader /data/AVA/csv/train_loader.csv \
  --visual-root /data/AVA/clips_videos \
  --audio-root /data/AVA/clips_audios \
  --split train          # writes /data/AVA/packed/train
```

The adapter reads nested JPEG face directories and clip-local WAV files.
Missing inputs fail preprocessing. `--skip-missing` exists only for deliberately
partial development copies.

## WASD

WASD ships the same layout and the same loader schema as AVA, so it uses the
same reader:

```bash
python -m preprocess.asd.prepare wasd \
  --loader /data/WASD/csv/train_loader.csv \
  --visual-root /data/WASD/clips_videos \
  --audio-root /data/WASD/clips_audios \
  --split train          # writes /data/WASD/packed/train
```

Two differences from AVA are worth knowing when training on both. WASD labels
are strictly binary - `SPEAKING_NOT_AUDIBLE` never occurs - so the rule that
inaudible speech counts as negative is something only AVA teaches. And every
face track in a WASD clip shares one clip-level WAV, so audio alone cannot tell
its speakers apart; any augmentation that mixes in a second voice must draw it
from a different clip.

## MSDWild

Create 30-second native-FPS clips with 10 seconds of overlap directly from the
paired raw MP4 and bbox CSV files. The first window starts at the first frame
containing any face. Tracks must be present in at least 90% of a complete
window, which removes transient detections before face cropping.
Incomplete final windows are discarded rather than padded, because the model
does not use a padding mask and would otherwise learn from artificial silence.
Speaking labels come from `all.rttm`, which ships separately from the
bounding-box release and must be downloaded before this script can run; the
final bbox CSV column is not an activity label. `prepare.py` has no MSDWild
path - there is no loader TSV to read.

```bash
python preprocess/asd/prep_msdwild.py \
  --input /data/MSDWild/msdwild_boundingbox_labels \
  --output /data/MSDWild/preprocessed
```

The script uses the shared audio loader and face cropper and writes the same
canonical memory-mapped shards as AVA and WASD. No intermediate per-clip files
or loader CSV are created.

## AVCC

AVCC uses the canonical headered TSV because conversation corpora vary in
their crop-generation layout:

```text
sample_id\tvisual_path\taudio_path\tsource_fps\tlabels\tstart_sec\tvisual_kind\taudio_sample_rate
conv1_spk1\tfaces/conv1_spk1.npy\taudio/conv1.wav\t25.0\t[0,1,1,0]\t12.0\tnpy\t16000
```

Paths may be absolute or relative to `--root`:

```bash
python -m preprocess.asd.prepare avcc \
  --loader /data/AVCC/asd_manifest.tsv \
  --root /data/AVCC \
  --output /data/packed/avcc_train
```

Each row represents one speaker face track. AVCC conversation grouping,
bounding boxes, and speaker identities remain separate MuVAP metadata; its ASD
embeddings must be extracted from these canonical samples.

## Lightning

```yaml
packed_asd:
  train:
    - /scratch/local/muvap/AVA/packed/train
    - /scratch/local/muvap/WASD/packed/train
    - /scratch/local/muvap/WASD/packed/val
  val:
    - /scratch/local/muvap/AVA/packed/val
```

Use `PackedASDDataModule`. Start with four workers per GPU,
`persistent_workers=True`, `prefetch_factor=2`, and pinned memory, then measure
0/2/4/8 workers on the actual allocation.

### Training batches

Track lengths are wildly uneven - AVA's median is 64 frames against WASD's 746 -
and the collator crops a batch to its shortest member. Batching at random
therefore threw away most of the data: measured on the AVA pack, a shuffled
`batch_size: 8` used **36%** of its frames per epoch.

Two settings fix that, and they only work together:

- `train_window` splits any track longer than it into near-equal chunks, so one
  epoch still covers every frame and no chunk is much longer than another.
- `frame_budget` builds each batch from similar-length chunks up to a target
  `batch * frames`, clamped by `min_batch` and `max_batch`.

Together they reach **99.95%** frame utilisation on AVA and 99.98% on AVA plus
WASD, which is why the model still needs no padding mask. Holding
`batch * frames` roughly constant also stabilises activation memory, the `B*T`
population the visual frontend's batch norm sees, and the per-frame gradient
noise of the mean-reduced losses. The clamp exists because the contrastive term
is a `B`-way classification whose difficulty would otherwise swing with the
chunk length.

### Context prefix

A chunk boundary creates a second, subtler problem. The model is causal, so a
frame near the start of a chunk can only attend to the few frames before it -
yet its projection target depends on 2 s of history. Without help, 31.6% of
scored frames on AVA plus WASD are asked to predict from history the encoder was
never shown.

The loader therefore reads `hist_frames` extra frames *before* each chunk and
marks them unscored. They provide history and never contribute to a loss, so:

- every frame is still scored exactly once per epoch,
- the artificial starvation disappears entirely - the remaining 21.5% are all
  real track starts, where the missing history is genuine and the model has to
  cope with it at inference too,
- the cost is 1.15x compute and no extra storage.

Overlapping the chunks instead would duplicate the data on disk *and* keep
training on the truncated copies, since it adds well-formed frames beside the
starved ones rather than removing them.

Validation ignores all of this: it runs whole tracks at batch size one, so every
native frame receives a score.

Because the train loader shards itself across ranks, `Trainer` is constructed
with `use_distributed_sampler=False` and validation gets an explicit
`DistributedSampler`.

Validate every completed dataset and compare amplitude distributions before
training:

```bash
python -m preprocess.asd.validate /data/AVA/packed/train /data/WASD/packed/train
```

The report shows the stored source levels beside the levels the loader produces
after normalization. Large source differences between corpora are expected;
large differences in the *normalized* RMS usually mean peak limiting,
near-silence, or malformed input that should be inspected.