Add causal person fusion and MuVAP stage-3 baselines

This commit is contained in:
Xin Wang
2026-09-10 06:54:22 +08:00
parent 0534c9d3d0
commit d426b4740a
225 changed files with 27716 additions and 119 deletions
+37
View File
@@ -73,6 +73,40 @@ result.
See the immutable
[`result record`](results/easycom_track_gru_residual_4ep_2026-09-07/README.md)
and [`implementation note`](docs/easycom_track_gru_residual.md).
- `causal-person-adapter-v2` is now implemented but unmeasured. It preserves an
exact matched pooled `G+A+V` base, reuses the same causal anonymous track
histories, replaces the blocking zero scalar with a sample-dependent vector
gate, and zero-initializes only the final residual-logit projection. Its
isolated runbook is
[`docs/easycom_causal_person_adapter_v2.md`](docs/easycom_causal_person_adapter_v2.md).
This is an optimization-repair/incremental screen, not evidence that track
identity matters.
- `causal-person-adapter-egospeak-gru` is an unmeasured controlled ablation of
V2. It changes only the per-track visual temporal stem to EgoSpeak
MiniROADA's `Linear-LN-ReLU-Dropout-GRU-ReLU` ordering while retaining V2's
pooled base, G+A person query, set attention, dynamic gate and zero-output
residual. It is an EgoSpeak-style per-track adaptation, not a reproduction of
EgoSpeak's global 5 Hz RGB+audio GRU. See its
[`runbook`](docs/easycom_egospeak_style_person_gru.md).
- `highrate-person-av-attention` is now implemented but unmeasured. A separate
extractor runs the published, frozen MuVAP frontend at 25 Hz: mixed audio is
encoded once as a shared CPC stream, while each face track contributes a
causal visual state plus one AV active-speaker score. At each 5 Hz right edge
`G+U` queries that variable person set. The first screen loads and freezes
the matched trained global-Omni + traditional-audio base and learns a
zero-initialized person-logit residual. This is global Omni context plus
traditional high-rate audio/person perception;
it is not the full MuVAP forecasting model and does not yet include online
detection. See the
[`runbook`](docs/easycom_highrate_person_av.md).
- A MuVAP-only Stage-3 baseline is now implemented as a clean-room paper
reconstruction. It freezes the released 25 Hz mono role-VAP and causal ASD
backbones, lets `Z_VAP` query the variable `Z_ASD` person set at each frame,
applies a one-layer causal temporal Transformer, and trains only that fusion
plus the EgoSpeak 10-horizon head. It deliberately excludes Omni and is not
described as upstream Stage-3 code because that entry point remains
unpublished. See the
[`runbook`](docs/easycom_muvap_stage3_baseline.md).
### Measured oracle-information screen
@@ -305,3 +339,6 @@ Do not reorganize them physically without compatibility wrappers.
- Remote paths differ (`/data/...` and `/data1/...`). Commands should expose
dataset/model roots as variables rather than hard-code one machine.
- Large models, datasets, checkpoints and extracted features live outside Git.
The explicitly audited 72 MiB MuVAP ASD-CPC release and 7 MiB CPC frontend
are a deliberate exception: they are vendored with their exact source and
provenance under `third_party/MuVAP` for reproducible extraction.