Add causal person fusion and MuVAP stage-3 baselines
This commit is contained in:
@@ -0,0 +1,113 @@
|
||||
# EasyCom causal person adapter V2
|
||||
|
||||
## Status and question
|
||||
|
||||
Implemented and CPU-testable as `--local-fusion causal-person-adapter-v2`;
|
||||
GPU results have not yet been measured. The mode is an optimization-corrected,
|
||||
incremental experiment after the matched pooled `G+A+V` baseline. It does not
|
||||
yet establish that track identity or visual grounding improves wearer floor
|
||||
behavior.
|
||||
|
||||
The experiment asks whether strictly causal, per-track visual histories contain
|
||||
information beyond equal person/frame pooling. It deliberately preserves the
|
||||
legacy `track-gru-residual` mode so its earlier null screen remains reproducible.
|
||||
The controlled `causal-person-adapter-egospeak-gru` ablation keeps this complete
|
||||
V2 fusion path and changes only the per-track temporal stem; see
|
||||
[`easycom_egospeak_style_person_gru.md`](easycom_egospeak_style_person_gru.md).
|
||||
|
||||
## Architecture
|
||||
|
||||
For 5 Hz row `t`, the media cutoff is `(t+1)/5` seconds. The local loader selects
|
||||
only person frames strictly earlier than that cutoff. Each currently visible
|
||||
anonymous track contributes at most `K` observations:
|
||||
|
||||
```text
|
||||
p_i,t = shared_GRU(project(v_i,t-K+1:t))
|
||||
q_t = project(G_t) + project(A_t)
|
||||
c_t = MHA(q_t, [NULL, p_1,t, ..., p_N,t])
|
||||
l_t = MLP([c_t, projected_A_t, person_valid, audio_valid])
|
||||
gate = sigmoid(MLP([pooled_base_hidden_t, l_t]))
|
||||
logits_t = pooled_logits_t + person_valid * W_delta(gate * l_t)
|
||||
```
|
||||
|
||||
`W_delta.weight` and `W_delta.bias` are zero initialized. After copying a
|
||||
matched pooled checkpoint, V2 therefore starts with logits exactly equal to
|
||||
the pooled model. The first backward pass updates `W_delta`; subsequent passes
|
||||
send gradients into the shared GRU, attention and dynamic gate. This fixes the
|
||||
legacy mode's zero scalar gate, which blocked all upstream branch gradients at
|
||||
initialization.
|
||||
|
||||
Track IDs are routing metadata only and never become embeddings. Parameters are
|
||||
shared across people, the attention uses a real-person mask plus a learned NULL
|
||||
person, source-clip/max-gap boundaries reset links, and changing padded values
|
||||
or permuting the current person order cannot change predictions. At the 2 FPS
|
||||
local grid, this runbook uses `K=4`, covering at most two seconds. Repeated 5 Hz
|
||||
queries
|
||||
recompute/read the same causal history; they do not append duplicate visual
|
||||
observations.
|
||||
|
||||
This immediate V2 keeps pooled visual evidence in the exact base. If it passes,
|
||||
a later replacement ablation must compare `G+A+person-track` without pooled V;
|
||||
otherwise the final method cannot claim unconditional pooling is unnecessary.
|
||||
|
||||
## L20 smoke test
|
||||
|
||||
The smoke test uses only training sessions and writes isolated outputs:
|
||||
|
||||
```bash
|
||||
EPOCHS=1 \
|
||||
WARMUP_EPOCHS=0 \
|
||||
SESSIONS="4 5" \
|
||||
DEV_SESSIONS=5 \
|
||||
SCREEN_SESSIONS=5 \
|
||||
CHECKPOINT_ROOT=checkpoints/easycom_person_adapter_v2_smoke \
|
||||
RESULT_ROOT=results/easycom_person_adapter_v2_smoke \
|
||||
CUDA_VISIBLE_DEVICES=0 \
|
||||
bash run_easycom_causal_person_adapter_v2.sh all 0
|
||||
```
|
||||
|
||||
Confirm that:
|
||||
|
||||
- the pooled checkpoint is accepted as an exact base;
|
||||
- `person_adapter_diagnostics.residual_output_weight_norm` becomes nonzero;
|
||||
- `delta_logit_mean_abs` becomes nonzero;
|
||||
- shapes and valid counts match the pooled run;
|
||||
- no Session 1--3 output is produced.
|
||||
|
||||
## Three-seed Session-12 screen
|
||||
|
||||
Run one seed per GPU in separate foreground terminals. Filenames are disjoint,
|
||||
so the runs may share the same roots:
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0 bash run_easycom_causal_person_adapter_v2.sh all 0
|
||||
CUDA_VISIBLE_DEVICES=1 bash run_easycom_causal_person_adapter_v2.sh all 1
|
||||
CUDA_VISIBLE_DEVICES=2 bash run_easycom_causal_person_adapter_v2.sh all 2
|
||||
```
|
||||
|
||||
Summarize after all processes finish:
|
||||
|
||||
```bash
|
||||
python summarize_local_baselines.py \
|
||||
--root results/easycom_causal_person_adapter_v2 \
|
||||
--dataset easycom \
|
||||
--protocol causal_valid_only \
|
||||
--paired-deltas
|
||||
```
|
||||
|
||||
Do not run the official test split while choosing this architecture. Once the
|
||||
method, seed-independent settings and event protocol are frozen, test inference
|
||||
is explicit:
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0 \
|
||||
bash run_easycom_causal_person_adapter_v2.sh test 0 1 2
|
||||
```
|
||||
|
||||
## Interpretation boundary
|
||||
|
||||
Dense AP/mAP is an architecture screen. The event evaluator still requires
|
||||
competing-risk censoring before response/yield metrics become the paper
|
||||
protocol. A publishable visual claim additionally requires correct-person,
|
||||
past-shifted, wrong-clip, NULL-person and parameter-matched controls. V2 alone
|
||||
does not predict active-speaker identity or addressee state.
|
||||
Reference in New Issue
Block a user