Add causal person fusion and MuVAP stage-3 baselines
This commit is contained in:
@@ -73,6 +73,40 @@ result.
|
||||
See the immutable
|
||||
[`result record`](results/easycom_track_gru_residual_4ep_2026-09-07/README.md)
|
||||
and [`implementation note`](docs/easycom_track_gru_residual.md).
|
||||
- `causal-person-adapter-v2` is now implemented but unmeasured. It preserves an
|
||||
exact matched pooled `G+A+V` base, reuses the same causal anonymous track
|
||||
histories, replaces the blocking zero scalar with a sample-dependent vector
|
||||
gate, and zero-initializes only the final residual-logit projection. Its
|
||||
isolated runbook is
|
||||
[`docs/easycom_causal_person_adapter_v2.md`](docs/easycom_causal_person_adapter_v2.md).
|
||||
This is an optimization-repair/incremental screen, not evidence that track
|
||||
identity matters.
|
||||
- `causal-person-adapter-egospeak-gru` is an unmeasured controlled ablation of
|
||||
V2. It changes only the per-track visual temporal stem to EgoSpeak
|
||||
MiniROADA's `Linear-LN-ReLU-Dropout-GRU-ReLU` ordering while retaining V2's
|
||||
pooled base, G+A person query, set attention, dynamic gate and zero-output
|
||||
residual. It is an EgoSpeak-style per-track adaptation, not a reproduction of
|
||||
EgoSpeak's global 5 Hz RGB+audio GRU. See its
|
||||
[`runbook`](docs/easycom_egospeak_style_person_gru.md).
|
||||
- `highrate-person-av-attention` is now implemented but unmeasured. A separate
|
||||
extractor runs the published, frozen MuVAP frontend at 25 Hz: mixed audio is
|
||||
encoded once as a shared CPC stream, while each face track contributes a
|
||||
causal visual state plus one AV active-speaker score. At each 5 Hz right edge
|
||||
`G+U` queries that variable person set. The first screen loads and freezes
|
||||
the matched trained global-Omni + traditional-audio base and learns a
|
||||
zero-initialized person-logit residual. This is global Omni context plus
|
||||
traditional high-rate audio/person perception;
|
||||
it is not the full MuVAP forecasting model and does not yet include online
|
||||
detection. See the
|
||||
[`runbook`](docs/easycom_highrate_person_av.md).
|
||||
- A MuVAP-only Stage-3 baseline is now implemented as a clean-room paper
|
||||
reconstruction. It freezes the released 25 Hz mono role-VAP and causal ASD
|
||||
backbones, lets `Z_VAP` query the variable `Z_ASD` person set at each frame,
|
||||
applies a one-layer causal temporal Transformer, and trains only that fusion
|
||||
plus the EgoSpeak 10-horizon head. It deliberately excludes Omni and is not
|
||||
described as upstream Stage-3 code because that entry point remains
|
||||
unpublished. See the
|
||||
[`runbook`](docs/easycom_muvap_stage3_baseline.md).
|
||||
|
||||
### Measured oracle-information screen
|
||||
|
||||
@@ -305,3 +339,6 @@ Do not reorganize them physically without compatibility wrappers.
|
||||
- Remote paths differ (`/data/...` and `/data1/...`). Commands should expose
|
||||
dataset/model roots as variables rather than hard-code one machine.
|
||||
- Large models, datasets, checkpoints and extracted features live outside Git.
|
||||
The explicitly audited 72 MiB MuVAP ASD-CPC release and 7 MiB CPC frontend
|
||||
are a deliberate exception: they are vendored with their exact source and
|
||||
provenance under `third_party/MuVAP` for reproducible extraction.
|
||||
|
||||
Reference in New Issue
Block a user