icassp2027-experiment/third_party/MuVAP/docs/preprocessing.md

Canonical ASD preprocessing

All datasets are converted to the same versioned muvap-asd-mmap format. Dataset adapters only resolve source metadata; preprocess.asd.writer performs all media conversion and alignment.

Packed format

A pack is a decoded mirror of the source corpus, not a model-ready tensor set. Faces and labels keep the native frame rate, audio keeps its full duration and its original gain. Everything MuVAP+ specific - sampling onto the 25 Hz model timeline, loudness normalization, chunking - happens in data.dataloaders.packed when a batch is built. That split is what lets a pack serve a model whose conventions differ from this one.

Each corpus keeps its pack beside its own media, so --output defaults to <dataset root>/packed/<split> and a dataset stays self-contained:

AVA/packed/train/
  dataset.json
  shard-000000/
    faces.npy   # uint8 [native_frames, 112, 112], grayscale
    audio.npy   # int16 mono 16 kHz, unnormalized, full clip duration
    labels.npy  # uint8, one label per NATIVE frame
    index.json  # offsets, source_fps, source_frames, audio level statistics

Each shard is approximately 1 GB. The arrays are uncompressed and opened with mmap_mode="r", so workers and DDP ranks share the operating-system page cache without HDF5 handles or a per-process copy. Copy complete dataset directories to node-local storage before training if the shared filesystem is the bottleneck.

The per-sample invariant is:

source_frames faces == source_frames labels
audio_samples == round(source_frames / source_fps * 16000)

What the reader adds

index.json carries source_fps, source_frames, and the recording's audio_mean, audio_rms, and audio_peak measured after DC removal. Those statistics are what let the loader reproduce a whole-recording normalization from any slice: gain computed from a chunk alone would drift with that chunk's content. The loader removes DC, scales to -25 dBFS RMS, caps gain at 20 dB and the peak at 0.99, and samples faces and labels onto the 25 Hz timeline with one shared index so a label stays attached to the face it describes.

The one convention still baked into the pack is the face crop: 112x112 grayscale, recorded in dataset.json as face_shape and face_color. It is the ASD community's standard geometry rather than a MuVAP+ choice, but a reader should check it rather than assume it.

The size is a constant in two places, not a config key: FACE_SIZE in preprocess/asd/schema.py, which the writer resizes every JPEG face to, and VisualSpec in data/media.py, which is what the loader and the model expect. Changing the geometry means editing both and re-running preprocessing - a pack written at one size cannot be read at another.

AVA

python -m preprocess.asd.prepare ava \
  --loader /data/AVA/csv/train_loader.csv \
  --visual-root /data/AVA/clips_videos \
  --audio-root /data/AVA/clips_audios \
  --split train          # writes /data/AVA/packed/train

The adapter reads nested JPEG face directories and clip-local WAV files. Missing inputs fail preprocessing. --skip-missing exists only for deliberately partial development copies.

WASD

WASD ships the same layout and the same loader schema as AVA, so it uses the same reader:

python -m preprocess.asd.prepare wasd \
  --loader /data/WASD/csv/train_loader.csv \
  --visual-root /data/WASD/clips_videos \
  --audio-root /data/WASD/clips_audios \
  --split train          # writes /data/WASD/packed/train

Two differences from AVA are worth knowing when training on both. WASD labels are strictly binary - SPEAKING_NOT_AUDIBLE never occurs - so the rule that inaudible speech counts as negative is something only AVA teaches. And every face track in a WASD clip shares one clip-level WAV, so audio alone cannot tell its speakers apart; any augmentation that mixes in a second voice must draw it from a different clip.

MSDWild

Create 30-second native-FPS clips with 10 seconds of overlap directly from the paired raw MP4 and bbox CSV files. The first window starts at the first frame containing any face. Tracks must be present in at least 90% of a complete window, which removes transient detections before face cropping. Incomplete final windows are discarded rather than padded, because the model does not use a padding mask and would otherwise learn from artificial silence. Speaking labels come from all.rttm, which ships separately from the bounding-box release and must be downloaded before this script can run; the final bbox CSV column is not an activity label. prepare.py has no MSDWild path - there is no loader TSV to read.

python preprocess/asd/prep_msdwild.py \
  --input /data/MSDWild/msdwild_boundingbox_labels \
  --output /data/MSDWild/preprocessed

The script uses the shared audio loader and face cropper and writes the same canonical memory-mapped shards as AVA and WASD. No intermediate per-clip files or loader CSV are created.

AVCC

AVCC uses the canonical headered TSV because conversation corpora vary in their crop-generation layout:

sample_id\tvisual_path\taudio_path\tsource_fps\tlabels\tstart_sec\tvisual_kind\taudio_sample_rate
conv1_spk1\tfaces/conv1_spk1.npy\taudio/conv1.wav\t25.0\t[0,1,1,0]\t12.0\tnpy\t16000

Paths may be absolute or relative to --root:

python -m preprocess.asd.prepare avcc \
  --loader /data/AVCC/asd_manifest.tsv \
  --root /data/AVCC \
  --output /data/packed/avcc_train

Each row represents one speaker face track. AVCC conversation grouping, bounding boxes, and speaker identities remain separate MuVAP metadata; its ASD embeddings must be extracted from these canonical samples.

Lightning

packed_asd:
  train:
    - /scratch/local/muvap/AVA/packed/train
    - /scratch/local/muvap/WASD/packed/train
    - /scratch/local/muvap/WASD/packed/val
  val:
    - /scratch/local/muvap/AVA/packed/val

Use PackedASDDataModule. Start with four workers per GPU, persistent_workers=True, prefetch_factor=2, and pinned memory, then measure 0/2/4/8 workers on the actual allocation.

Training batches

Track lengths are wildly uneven - AVA's median is 64 frames against WASD's 746 - and the collator crops a batch to its shortest member. Batching at random therefore threw away most of the data: measured on the AVA pack, a shuffled batch_size: 8 used 36% of its frames per epoch.

Two settings fix that, and they only work together:

• train_window splits any track longer than it into near-equal chunks, so one epoch still covers every frame and no chunk is much longer than another.
• frame_budget builds each batch from similar-length chunks up to a target batch * frames, clamped by min_batch and max_batch.

Together they reach 99.95% frame utilisation on AVA and 99.98% on AVA plus WASD, which is why the model still needs no padding mask. Holding batch * frames roughly constant also stabilises activation memory, the B*T population the visual frontend's batch norm sees, and the per-frame gradient noise of the mean-reduced losses. The clamp exists because the contrastive term is a B-way classification whose difficulty would otherwise swing with the chunk length.

Context prefix

A chunk boundary creates a second, subtler problem. The model is causal, so a frame near the start of a chunk can only attend to the few frames before it - yet its projection target depends on 2 s of history. Without help, 31.6% of scored frames on AVA plus WASD are asked to predict from history the encoder was never shown.

The loader therefore reads hist_frames extra frames before each chunk and marks them unscored. They provide history and never contribute to a loss, so:

• every frame is still scored exactly once per epoch,
• the artificial starvation disappears entirely - the remaining 21.5% are all real track starts, where the missing history is genuine and the model has to cope with it at inference too,
• the cost is 1.15x compute and no extra storage.

Overlapping the chunks instead would duplicate the data on disk and keep training on the truncated copies, since it adds well-formed frames beside the starved ones rather than removing them.

Validation ignores all of this: it runs whole tracks at batch size one, so every native frame receives a score.

Because the train loader shards itself across ranks, Trainer is constructed with use_distributed_sampler=False and validation gets an explicit DistributedSampler.

Validate every completed dataset and compare amplitude distributions before training:

python -m preprocess.asd.validate /data/AVA/packed/train /data/WASD/packed/train

The report shows the stored source levels beside the levels the loader produces after normalization. Large source differences between corpora are expected; large differences in the normalized RMS usually mean peak limiting, near-silence, or malformed input that should be inspected.