# Canonical ASD preprocessing All datasets are converted to the same versioned `muvap-asd-mmap` format. Dataset adapters only resolve source metadata; `preprocess.asd.writer` performs all media conversion and alignment. ## Packed format A pack is a **decoded mirror of the source corpus, not a model-ready tensor set**. Faces and labels keep the native frame rate, audio keeps its full duration and its original gain. Everything MuVAP+ specific - sampling onto the 25 Hz model timeline, loudness normalization, chunking - happens in `data.dataloaders.packed` when a batch is built. That split is what lets a pack serve a model whose conventions differ from this one. Each corpus keeps its pack beside its own media, so `--output` defaults to `/packed/` and a dataset stays self-contained: ```text AVA/packed/train/ dataset.json shard-000000/ faces.npy # uint8 [native_frames, 112, 112], grayscale audio.npy # int16 mono 16 kHz, unnormalized, full clip duration labels.npy # uint8, one label per NATIVE frame index.json # offsets, source_fps, source_frames, audio level statistics ``` Each shard is approximately 1 GB. The arrays are uncompressed and opened with `mmap_mode="r"`, so workers and DDP ranks share the operating-system page cache without HDF5 handles or a per-process copy. Copy complete dataset directories to node-local storage before training if the shared filesystem is the bottleneck. The per-sample invariant is: ```text source_frames faces == source_frames labels audio_samples == round(source_frames / source_fps * 16000) ``` ### What the reader adds `index.json` carries `source_fps`, `source_frames`, and the recording's `audio_mean`, `audio_rms`, and `audio_peak` measured after DC removal. Those statistics are what let the loader reproduce a whole-recording normalization from any slice: gain computed from a chunk alone would drift with that chunk's content. The loader removes DC, scales to -25 dBFS RMS, caps gain at 20 dB and the peak at 0.99, and samples faces and labels onto the 25 Hz timeline with one shared index so a label stays attached to the face it describes. The one convention still baked into the pack is the face crop: 112x112 grayscale, recorded in `dataset.json` as `face_shape` and `face_color`. It is the ASD community's standard geometry rather than a MuVAP+ choice, but a reader should check it rather than assume it. The size is a constant in two places, not a config key: `FACE_SIZE` in `preprocess/asd/schema.py`, which the writer resizes every JPEG face to, and `VisualSpec` in `data/media.py`, which is what the loader and the model expect. Changing the geometry means editing both and re-running preprocessing - a pack written at one size cannot be read at another. ## AVA ```bash python -m preprocess.asd.prepare ava \ --loader /data/AVA/csv/train_loader.csv \ --visual-root /data/AVA/clips_videos \ --audio-root /data/AVA/clips_audios \ --split train # writes /data/AVA/packed/train ``` The adapter reads nested JPEG face directories and clip-local WAV files. Missing inputs fail preprocessing. `--skip-missing` exists only for deliberately partial development copies. ## WASD WASD ships the same layout and the same loader schema as AVA, so it uses the same reader: ```bash python -m preprocess.asd.prepare wasd \ --loader /data/WASD/csv/train_loader.csv \ --visual-root /data/WASD/clips_videos \ --audio-root /data/WASD/clips_audios \ --split train # writes /data/WASD/packed/train ``` Two differences from AVA are worth knowing when training on both. WASD labels are strictly binary - `SPEAKING_NOT_AUDIBLE` never occurs - so the rule that inaudible speech counts as negative is something only AVA teaches. And every face track in a WASD clip shares one clip-level WAV, so audio alone cannot tell its speakers apart; any augmentation that mixes in a second voice must draw it from a different clip. ## MSDWild Create 30-second native-FPS clips with 10 seconds of overlap directly from the paired raw MP4 and bbox CSV files. The first window starts at the first frame containing any face. Tracks must be present in at least 90% of a complete window, which removes transient detections before face cropping. Incomplete final windows are discarded rather than padded, because the model does not use a padding mask and would otherwise learn from artificial silence. Speaking labels come from `all.rttm`, which ships separately from the bounding-box release and must be downloaded before this script can run; the final bbox CSV column is not an activity label. `prepare.py` has no MSDWild path - there is no loader TSV to read. ```bash python preprocess/asd/prep_msdwild.py \ --input /data/MSDWild/msdwild_boundingbox_labels \ --output /data/MSDWild/preprocessed ``` The script uses the shared audio loader and face cropper and writes the same canonical memory-mapped shards as AVA and WASD. No intermediate per-clip files or loader CSV are created. ## AVCC AVCC uses the canonical headered TSV because conversation corpora vary in their crop-generation layout: ```text sample_id\tvisual_path\taudio_path\tsource_fps\tlabels\tstart_sec\tvisual_kind\taudio_sample_rate conv1_spk1\tfaces/conv1_spk1.npy\taudio/conv1.wav\t25.0\t[0,1,1,0]\t12.0\tnpy\t16000 ``` Paths may be absolute or relative to `--root`: ```bash python -m preprocess.asd.prepare avcc \ --loader /data/AVCC/asd_manifest.tsv \ --root /data/AVCC \ --output /data/packed/avcc_train ``` Each row represents one speaker face track. AVCC conversation grouping, bounding boxes, and speaker identities remain separate MuVAP metadata; its ASD embeddings must be extracted from these canonical samples. ## Lightning ```yaml packed_asd: train: - /scratch/local/muvap/AVA/packed/train - /scratch/local/muvap/WASD/packed/train - /scratch/local/muvap/WASD/packed/val val: - /scratch/local/muvap/AVA/packed/val ``` Use `PackedASDDataModule`. Start with four workers per GPU, `persistent_workers=True`, `prefetch_factor=2`, and pinned memory, then measure 0/2/4/8 workers on the actual allocation. ### Training batches Track lengths are wildly uneven - AVA's median is 64 frames against WASD's 746 - and the collator crops a batch to its shortest member. Batching at random therefore threw away most of the data: measured on the AVA pack, a shuffled `batch_size: 8` used **36%** of its frames per epoch. Two settings fix that, and they only work together: - `train_window` splits any track longer than it into near-equal chunks, so one epoch still covers every frame and no chunk is much longer than another. - `frame_budget` builds each batch from similar-length chunks up to a target `batch * frames`, clamped by `min_batch` and `max_batch`. Together they reach **99.95%** frame utilisation on AVA and 99.98% on AVA plus WASD, which is why the model still needs no padding mask. Holding `batch * frames` roughly constant also stabilises activation memory, the `B*T` population the visual frontend's batch norm sees, and the per-frame gradient noise of the mean-reduced losses. The clamp exists because the contrastive term is a `B`-way classification whose difficulty would otherwise swing with the chunk length. ### Context prefix A chunk boundary creates a second, subtler problem. The model is causal, so a frame near the start of a chunk can only attend to the few frames before it - yet its projection target depends on 2 s of history. Without help, 31.6% of scored frames on AVA plus WASD are asked to predict from history the encoder was never shown. The loader therefore reads `hist_frames` extra frames *before* each chunk and marks them unscored. They provide history and never contribute to a loss, so: - every frame is still scored exactly once per epoch, - the artificial starvation disappears entirely - the remaining 21.5% are all real track starts, where the missing history is genuine and the model has to cope with it at inference too, - the cost is 1.15x compute and no extra storage. Overlapping the chunks instead would duplicate the data on disk *and* keep training on the truncated copies, since it adds well-formed frames beside the starved ones rather than removing them. Validation ignores all of this: it runs whole tracks at batch size one, so every native frame receives a score. Because the train loader shards itself across ranks, `Trainer` is constructed with `use_distributed_sampler=False` and validation gets an explicit `DistributedSampler`. Validate every completed dataset and compare amplitude distributions before training: ```bash python -m preprocess.asd.validate /data/AVA/packed/train /data/WASD/packed/train ``` The report shows the stored source levels beside the levels the loader produces after normalization. Large source differences between corpora are expected; large differences in the *normalized* RMS usually mean peak limiting, near-silence, or malformed input that should be inspected.