icassp2027-experiment/docs/easycom_egospeak_style_person_gru.md

EasyCom EgoSpeak-style person-track GRU ablation

Status and purpose

Implemented as --local-fusion causal-person-adapter-egospeak-gru; GPU results are not yet measured. This is a controlled temporal-stem ablation of causal-person-adapter-v2, not a reproduction of EgoSpeak MiniROADA.

The comparison changes only how each anonymous person's causal visual history is embedded before the existing GRU/set-attention path:

Component	Native V2	EgoSpeak-style ablation
track input	LayerNorm -> Linear -> GELU	Linear -> LayerNorm -> ReLU -> Dropout
temporal layer	one-layer shared causal GRU	the same one-layer shared causal GRU
GRU output	final hidden state	ReLU(final hidden state)
all downstream fusion	unchanged	unchanged

Both modes use the same raw local visual features, causal track histories, history length, G+A query, masked multi-person attention, direct local-audio state, dynamic vector gate, zero-initialized residual output, pooled base checkpoint, loss, split, optimizer and seed. The actual trainable parameter count is printed and stored because moving LayerNorm across the projection causes a small parameter-count difference. With the default 2048D visual input and 256D person state, the EgoSpeak-style stem has 3,584 fewer affine LayerNorm parameters than native V2; all Linear and GRU matrix shapes match.

Relation to EgoSpeak

EgoSpeak MiniROADA applies Linear -> LayerNorm -> ReLU -> Dropout -> one-layer GRU -> ReLU to one global 5 Hz RGB+audio sequence. Its EasyCom configuration uses a 128-step window, 2048D embedding and 1024D hidden state, followed by its own anticipation expansion and shared classifier.

This repository uses a clean-room PyTorch implementation of only that temporal stem and applies it independently to each anonymous visual person track. It intentionally retains V2's 256D screen, two-second person history and residual forecasting head so the experiment isolates temporal parameterization. The correct name in results and the paper is EgoSpeak-style per-track GRU, not “EgoSpeak reproduction.” The audited reference is third_party/EgoSpeak/MiniROAD/model/rnn/rnn.py.

L20 smoke test

Use only training sessions and isolated output roots:

EPOCHS=1 \
WARMUP_EPOCHS=0 \
SESSIONS="4 5" \
DEV_SESSIONS=5 \
SCREEN_SESSIONS=5 \
CHECKPOINT_ROOT=checkpoints/easycom_egospeak_person_gru_smoke \
RESULT_ROOT=results/easycom_egospeak_person_gru_smoke \
CUDA_VISIBLE_DEVICES=0 \
bash run_easycom_causal_person_adapter_v2.sh all 0

This runs the matched pooled base, native V2 and EgoSpeak-style ablation. Check that all three use the same split and base-checkpoint hash, both adapter modes start from the pooled logits, and their residual-output/delta diagnostics become nonzero after training.

Three-seed development screen

If matched pooled/V2 checkpoints already exist, run only the new ablation:

SUMMARIZE=0 CUDA_VISIBLE_DEVICES=0 bash run_easycom_causal_person_adapter_v2.sh egospeak-gru 0
SUMMARIZE=0 CUDA_VISIBLE_DEVICES=1 bash run_easycom_causal_person_adapter_v2.sh egospeak-gru 1
SUMMARIZE=0 CUDA_VISIBLE_DEVICES=2 bash run_easycom_causal_person_adapter_v2.sh egospeak-gru 2

Otherwise, run all instead of egospeak-gru to create each matched pooled checkpoint and both adapters. Do not run official Sessions 1--3 while selecting the temporal stem.

After all jobs finish:

python summarize_local_baselines.py \
  --root results/easycom_causal_person_adapter_v2 \
  --dataset easycom \
  --protocol causal_valid_only \
  --paired-deltas

The relevant paired rows are egospeak_gru - pooled and egospeak_gru - person_adapter_v2. Dense AP/mAP is only the architecture screen; wearer event claims still wait for the frozen competing-risk evaluator.