114 lines
4.3 KiB
Markdown
114 lines
4.3 KiB
Markdown
|
# EasyCom causal person adapter V2
|
||
|
|
|
||
|
|
## Status and question
|
||
|
|
|
||
|
|
Implemented and CPU-testable as `--local-fusion causal-person-adapter-v2`;
|
||
|
|
GPU results have not yet been measured. The mode is an optimization-corrected,
|
||
|
|
incremental experiment after the matched pooled `G+A+V` baseline. It does not
|
||
|
|
yet establish that track identity or visual grounding improves wearer floor
|
||
|
|
behavior.
|
||
|
|
|
||
|
|
The experiment asks whether strictly causal, per-track visual histories contain
|
||
|
|
information beyond equal person/frame pooling. It deliberately preserves the
|
||
|
|
legacy `track-gru-residual` mode so its earlier null screen remains reproducible.
|
||
|
|
The controlled `causal-person-adapter-egospeak-gru` ablation keeps this complete
|
||
|
|
V2 fusion path and changes only the per-track temporal stem; see
|
||
|
|
[`easycom_egospeak_style_person_gru.md`](easycom_egospeak_style_person_gru.md).
|
||
|
|
|
||
|
|
## Architecture
|
||
|
|
|
||
|
|
For 5 Hz row `t`, the media cutoff is `(t+1)/5` seconds. The local loader selects
|
||
|
|
only person frames strictly earlier than that cutoff. Each currently visible
|
||
|
|
anonymous track contributes at most `K` observations:
|
||
|
|
|
||
|
|
```text
|
||
|
|
p_i,t = shared_GRU(project(v_i,t-K+1:t))
|
||
|
|
q_t = project(G_t) + project(A_t)
|
||
|
|
c_t = MHA(q_t, [NULL, p_1,t, ..., p_N,t])
|
||
|
|
l_t = MLP([c_t, projected_A_t, person_valid, audio_valid])
|
||
|
|
gate = sigmoid(MLP([pooled_base_hidden_t, l_t]))
|
||
|
|
logits_t = pooled_logits_t + person_valid * W_delta(gate * l_t)
|
||
|
|
```
|
||
|
|
|
||
|
|
`W_delta.weight` and `W_delta.bias` are zero initialized. After copying a
|
||
|
|
matched pooled checkpoint, V2 therefore starts with logits exactly equal to
|
||
|
|
the pooled model. The first backward pass updates `W_delta`; subsequent passes
|
||
|
|
send gradients into the shared GRU, attention and dynamic gate. This fixes the
|
||
|
|
legacy mode's zero scalar gate, which blocked all upstream branch gradients at
|
||
|
|
initialization.
|
||
|
|
|
||
|
|
Track IDs are routing metadata only and never become embeddings. Parameters are
|
||
|
|
shared across people, the attention uses a real-person mask plus a learned NULL
|
||
|
|
person, source-clip/max-gap boundaries reset links, and changing padded values
|
||
|
|
or permuting the current person order cannot change predictions. At the 2 FPS
|
||
|
|
local grid, this runbook uses `K=4`, covering at most two seconds. Repeated 5 Hz
|
||
|
|
queries
|
||
|
|
recompute/read the same causal history; they do not append duplicate visual
|
||
|
|
observations.
|
||
|
|
|
||
|
|
This immediate V2 keeps pooled visual evidence in the exact base. If it passes,
|
||
|
|
a later replacement ablation must compare `G+A+person-track` without pooled V;
|
||
|
|
otherwise the final method cannot claim unconditional pooling is unnecessary.
|
||
|
|
|
||
|
|
## L20 smoke test
|
||
|
|
|
||
|
|
The smoke test uses only training sessions and writes isolated outputs:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
EPOCHS=1 \
|
||
|
|
WARMUP_EPOCHS=0 \
|
||
|
|
SESSIONS="4 5" \
|
||
|
|
DEV_SESSIONS=5 \
|
||
|
|
SCREEN_SESSIONS=5 \
|
||
|
|
CHECKPOINT_ROOT=checkpoints/easycom_person_adapter_v2_smoke \
|
||
|
|
RESULT_ROOT=results/easycom_person_adapter_v2_smoke \
|
||
|
|
CUDA_VISIBLE_DEVICES=0 \
|
||
|
|
bash run_easycom_causal_person_adapter_v2.sh all 0
|
||
|
|
```
|
||
|
|
|
||
|
|
Confirm that:
|
||
|
|
|
||
|
|
- the pooled checkpoint is accepted as an exact base;
|
||
|
|
- `person_adapter_diagnostics.residual_output_weight_norm` becomes nonzero;
|
||
|
|
- `delta_logit_mean_abs` becomes nonzero;
|
||
|
|
- shapes and valid counts match the pooled run;
|
||
|
|
- no Session 1--3 output is produced.
|
||
|
|
|
||
|
|
## Three-seed Session-12 screen
|
||
|
|
|
||
|
|
Run one seed per GPU in separate foreground terminals. Filenames are disjoint,
|
||
|
|
so the runs may share the same roots:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
CUDA_VISIBLE_DEVICES=0 bash run_easycom_causal_person_adapter_v2.sh all 0
|
||
|
|
CUDA_VISIBLE_DEVICES=1 bash run_easycom_causal_person_adapter_v2.sh all 1
|
||
|
|
CUDA_VISIBLE_DEVICES=2 bash run_easycom_causal_person_adapter_v2.sh all 2
|
||
|
|
```
|
||
|
|
|
||
|
|
Summarize after all processes finish:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
python summarize_local_baselines.py \
|
||
|
|
--root results/easycom_causal_person_adapter_v2 \
|
||
|
|
--dataset easycom \
|
||
|
|
--protocol causal_valid_only \
|
||
|
|
--paired-deltas
|
||
|
|
```
|
||
|
|
|
||
|
|
Do not run the official test split while choosing this architecture. Once the
|
||
|
|
method, seed-independent settings and event protocol are frozen, test inference
|
||
|
|
is explicit:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
CUDA_VISIBLE_DEVICES=0 \
|
||
|
|
bash run_easycom_causal_person_adapter_v2.sh test 0 1 2
|
||
|
|
```
|
||
|
|
|
||
|
|
## Interpretation boundary
|
||
|
|
|
||
|
|
Dense AP/mAP is an architecture screen. The event evaluator still requires
|
||
|
|
competing-risk censoring before response/yield metrics become the paper
|
||
|
|
protocol. A publishable visual claim additionally requires correct-person,
|
||
|
|
past-shifted, wrong-clip, NULL-person and parameter-matched controls. V2 alone
|
||
|
|
does not predict active-speaker identity or addressee state.
|