icassp2027-experiment/docs/easycom_track_gru_residual.md

# EasyCom anonymous track-GRU residual

## Decision

The first track-temporal experiment keeps the strongest pooled local baseline
as an exact, optionally frozen base and learns only a causal person-track
correction:

```text
pooled base:  G_t + A_t + pooled-V_t -> concat-MLP -> base logits

per visible person i:
  past crop observations on the same temporary track
      -> shared projection -> shared one-layer GRU -> H_t,i

G_t + A_t -> masked attention over {NULL, H_t,1, ..., H_t,N}
          -> zero-gated residual logits

final logits = base logits + tanh(alpha) * person_valid * residual logits
alpha starts at exactly zero
```

This is the smallest experiment that asks whether anonymous visual continuity
adds information beyond `G+A+pooled-V`. It is not yet person-conditioned audio
grounding and does not modify the Omni Thinker.

## What was taken from EgoSpeak

The official EgoSpeak repository is cloned for inspection at
`third_party/EgoSpeak` and excluded from Git. The inspected revision is
`b5bee35e796d8f2d6aa7da2bf1b07836270f8bfb`.

Its `MiniROADA` model projects concatenated RGB/audio features, applies a
single-layer GRU, expands the recurrent state into ten anticipation states,
and uses a shared classifier. The EasyCom configuration uses a 128-step
window, stride 4, 2048D embedding, 1024D GRU and ten future steps.

We reuse the defensible ideas—causal recurrence, shared weights and a ten-step
anticipation output—but not the implementation. Our GRU is shared across
anonymous person tracks, reads only crop observations at or before the current
cutoff, and corrects the already measured pooled Omni baseline. Consequently,
this method must be described as an EgoSpeak-inspired track-temporal adapter,
not an EgoSpeak GRU reproduction.

## Causal and identity contract

- A 5 Hz row with cutoff `(t+1)/5` reads the latest local video frame strictly
  before that cutoff.
- Track history contains at most `--person-history-frames` real 2 FPS crop
  observations and is ordered oldest to newest.
- Temporary IDs only link observations. They are never embedded or passed to
  the model, so global ID renaming cannot change a prediction.
- A source-clip boundary always resets a track. A time gap larger than
  `--person-track-max-gap-seconds` also resets it.
- Missing, `-1`, `None`, `null` and `unknown` IDs are isolated observations.
- Duplicate known IDs in one frame fail instead of being silently merged.
- Empty person sets produce exactly zero visual correction.

The existing EasyCom local sidecars already contain crop features, frame times,
source-clip indices and person IDs; this experiment does not require feature
re-extraction.

Runtime status (2026-09-07): the focused causal-history, exact-zero-gate and
train/infer round-trip tests passed in 49.699 seconds on the remote PyTorch/L20
environment. This verifies the code path, not model quality. The one-session
GPU training smoke and full experiment remain unmeasured.

## Fair initialization

`track-gru-residual` requires `--initialize-pooled-checkpoint`. The loader
checks dataset, train/dev split, loss, layer fusion, dimensions and local
feature contract before copying the pooled path. By default those copied
parameters are frozen. The residual gate starts at zero, so before optimization
the track model must reproduce the pooled logits exactly, element for element.

The matched screening protocol is:

- EasyCom Session 4--11: training;
- Session 12: development/model selection;
- Session 1--3: final test, run only after history/gap and the method are frozen;
- 4 epochs, three seeds, identical optimizer/loss for pooled and track runs;
- 16 observations at 2 FPS (up to 8 seconds), 1-second maximum link gap.

This protocol intentionally re-runs the pooled base. Historical no-dev results
cannot be used as the matched control.

## Run

After pulling on the L20 machine, run the focused CPU regression tests first:

```bash
python -m unittest \
  test_local_turn_features.LocalTests.test_track_loader_builds_causal_histories_and_resets_tracks \
  test_local_turn_features.LocalTests.test_track_gru_residual_starts_from_exact_pooled_logits \
  test_local_turn_features.RoundTripTests.test_train_infer_easycom_and_ego4d
```

One-seed runtime smoke on an L20:

```bash
EPOCHS=1 \
SESSIONS="4 5" \
DEV_SESSIONS=5 \
SCREEN_SESSIONS=5 \
CHECKPOINT_ROOT=checkpoints/easycom_track_gru_smoke \
RESULT_ROOT=results/easycom_track_gru_smoke \
CUDA_VISIBLE_DEVICES=0 \
bash run_easycom_track_gru_residual.sh all 0
```

For `EPOCHS=1`, the launcher automatically uses zero warmup unless
`WARMUP_EPOCHS` is explicitly supplied.

Then use three GPUs in three foreground terminals. These runs report Session
12 development scores, not the official test set:

```bash
SUMMARIZE=0 CUDA_VISIBLE_DEVICES=0 bash run_easycom_track_gru_residual.sh all 0
SUMMARIZE=0 CUDA_VISIBLE_DEVICES=1 bash run_easycom_track_gru_residual.sh all 1
SUMMARIZE=0 CUDA_VISIBLE_DEVICES=2 bash run_easycom_track_gru_residual.sh all 2
```

After all development jobs finish:

```bash
python summarize_local_baselines.py \
  --root results/easycom_track_gru_residual \
  --dataset easycom \
  --protocol causal_valid_only \
  --paired-deltas
```

Only after choosing and freezing the method, evaluate the saved checkpoints
once on Sessions 1--3:

```bash
CUDA_VISIBLE_DEVICES=0 bash run_easycom_track_gru_residual.sh test 0 1 2
```

This writes and summarizes `results/easycom_track_gru_residual_test` without
retraining. A single GPU is sufficient because head inference is inexpensive.

Paths and hyperparameters can be overridden with the environment variables at
the top of `run_easycom_track_gru_residual.sh`. The launcher runs in the
foreground and refuses to overwrite an existing checkpoint or result.

## Interpretation and stopping rule

The initial screen answers only whether track-temporal visual structure can
improve the dense EgoSpeak-compatible metric. Advance this branch only if
same-seed `track_gru - pooled` deltas are directionally consistent on Session
12 without sacrificing short-horizon wearer AP. The official test is a final
confirmation, not another model-selection stage.
Because prior F1 variation was roughly a few tenths of a percentage point,
tiny single-seed changes are not evidence.

If the track branch wins, next run correct-track versus broken-track,
past-shifted and no-ID temporal controls, followed by person-conditioned audio
grounding. If it does not win, first compare 8/16/32 observations and optionally
unfreeze the pooled base using Session 12 only; do not add deeper cross-attention
merely to rescue a null result.

The paper-level claim still requires the frozen competing-risk event evaluator
and matched-recall false-entry analysis. Dense AP/mAP improvement alone is a
representation result, not proof that the model understands multi-person floor
behavior.