97 lines
4.0 KiB
Markdown
97 lines
4.0 KiB
Markdown
|
# Audio-visual data contract
|
||
|
|
|
||
|
|
## Canonical timeline
|
||
|
|
|
||
|
|
MuVAP+ uses a **25 Hz model timeline**, because the CPC stack emits one audio
|
||
|
|
embedding per 640 samples at 16 kHz. Original videos are not transcoded or
|
||
|
|
rewritten to 25 fps.
|
||
|
|
|
||
|
|
For model frame `k`, the timestamp is the centre of its interval:
|
||
|
|
|
||
|
|
```text
|
||
|
|
t(k) = (k + 0.5) / 25 seconds
|
||
|
|
```
|
||
|
|
|
||
|
|
The visual face and label are selected from the native source interval that
|
||
|
|
contains `t(k)`:
|
||
|
|
|
||
|
|
```text
|
||
|
|
source_index(k) = floor(t(k) * source_fps)
|
||
|
|
```
|
||
|
|
|
||
|
|
The exact same index is used for the face and its label. This prevents label
|
||
|
|
drift. At 30 fps some source frames are skipped; below 25 fps some are repeated.
|
||
|
|
|
||
|
|
**This happens in the loader, not in the pack.** Packed shards store the native
|
||
|
|
frame rate, so the sampling above is applied when a batch is built and can be
|
||
|
|
changed - or replaced with a different `model_fps` - without re-packing.
|
||
|
|
|
||
|
|
## Evaluation on the native timeline
|
||
|
|
|
||
|
|
Metrics defined on the source annotations — the official AVA mAP above all —
|
||
|
|
need one score per native frame, not per model frame. `model_indices` reverses
|
||
|
|
the mapping by asking which model interval contains each source-frame centre:
|
||
|
|
|
||
|
|
```text
|
||
|
|
model_index(j) = floor(((j + 0.5) / source_fps) * 25)
|
||
|
|
```
|
||
|
|
|
||
|
|
The result always has exactly `source_frames` entries, so every groundtruth row
|
||
|
|
receives a score at any frame rate. Below 25 fps this is an exact inverse of
|
||
|
|
`source_indices`. Above it, several source frames share a model frame, which is
|
||
|
|
information the 25 Hz timeline genuinely does not carry. Only the final source
|
||
|
|
frame can fall past the last complete model interval; it is clamped to that
|
||
|
|
interval, and no interior frame is ever mis-scored.
|
||
|
|
|
||
|
|
## Audio
|
||
|
|
|
||
|
|
Audio is loaded directly from WAV/FLAC (or NumPy waveforms), mixed to
|
||
|
|
mono, and resampled to 16 kHz for CPC. Packed shards keep the full span the face
|
||
|
|
frames cover, at the recording's own gain.
|
||
|
|
|
||
|
|
The loader takes it from there. A batch of `T` model frames always carries
|
||
|
|
exactly `T * 640` samples, right-padded with silence if the recording is short.
|
||
|
|
Loudness normalization to -25 dBFS RMS is applied at that point, using level
|
||
|
|
statistics measured over the **whole recording** and stored in the index -
|
||
|
|
computing gain from a chunk alone would make a sample's level depend on which
|
||
|
|
chunk it landed in.
|
||
|
|
|
||
|
|
## Required manifest fields
|
||
|
|
|
||
|
|
Each row must resolve to:
|
||
|
|
|
||
|
|
| Field | Meaning |
|
||
|
|
|---|---|
|
||
|
|
| `sample_id` | Stable, unique face-track/clip identifier |
|
||
|
|
| `audio_path` | Recording or clip audio path |
|
||
|
|
| `visual_path` | NumPy face array shaped `[source_frames, H, W]` |
|
||
|
|
| `source_fps` | Native FPS used by faces and labels |
|
||
|
|
| `labels` | One active-speaker label per native face frame |
|
||
|
|
| `start_sec` | Clip start on the audio recording timeline; default `0` only when audio is already clip-local |
|
||
|
|
|
||
|
|
The canonical file is a tab-separated text file with a header. `labels` is a
|
||
|
|
JSON array; paths may be absolute or relative to the manifest directory:
|
||
|
|
|
||
|
|
```text
|
||
|
|
sample_id\taudio_path\tvisual_path\tsource_fps\tlabels\tstart_sec
|
||
|
|
clip_001\taudio/recording.wav\tfaces/clip_001.npy\t29.97\t[0,0,1,1]\t12.48
|
||
|
|
```
|
||
|
|
|
||
|
|
`len(labels)` and the face-frame count must match. Preprocessing rejects a
|
||
|
|
sample where they disagree rather than trimming to the shorter one: a length
|
||
|
|
mismatch means the face track and the annotation disagree about the clip, which
|
||
|
|
would silently shift every later label.
|
||
|
|
|
||
|
|
## Dataset policy
|
||
|
|
|
||
|
|
- AVA, WASD, and MSDWild use the same alignment implementation.
|
||
|
|
- FPS is never ignored and must never be replaced with a global constant.
|
||
|
|
- Packs are decode-only. Any transform whose parameters belong to the model -
|
||
|
|
frame rate, loudness, crop duration - happens in the loader, so a pack stays
|
||
|
|
usable by code that does not share those conventions.
|
||
|
|
- Labels are aligned before projection-window targets are computed.
|
||
|
|
- Train-time batching may crop examples to a shared duration, but audio, faces,
|
||
|
|
VAD labels, and projected labels must all use that same duration.
|
||
|
|
- A WASD face track using full-recording audio must provide `start_sec`. It is
|
||
|
|
mathematically impossible to recover that offset from FPS and labels alone.
|