159 lines
6.3 KiB
Markdown
159 lines
6.3 KiB
Markdown
# EasyCom anonymous track-GRU residual
|
|
|
|
## Decision
|
|
|
|
The first track-temporal experiment keeps the strongest pooled local baseline
|
|
as an exact, optionally frozen base and learns only a causal person-track
|
|
correction:
|
|
|
|
```text
|
|
pooled base: G_t + A_t + pooled-V_t -> concat-MLP -> base logits
|
|
|
|
per visible person i:
|
|
past crop observations on the same temporary track
|
|
-> shared projection -> shared one-layer GRU -> H_t,i
|
|
|
|
G_t + A_t -> masked attention over {NULL, H_t,1, ..., H_t,N}
|
|
-> zero-gated residual logits
|
|
|
|
final logits = base logits + tanh(alpha) * person_valid * residual logits
|
|
alpha starts at exactly zero
|
|
```
|
|
|
|
This is the smallest experiment that asks whether anonymous visual continuity
|
|
adds information beyond `G+A+pooled-V`. It is not yet person-conditioned audio
|
|
grounding and does not modify the Omni Thinker.
|
|
|
|
## What was taken from EgoSpeak
|
|
|
|
The official EgoSpeak repository is cloned for inspection at
|
|
`third_party/EgoSpeak` and excluded from Git. The inspected revision is
|
|
`b5bee35e796d8f2d6aa7da2bf1b07836270f8bfb`.
|
|
|
|
Its `MiniROADA` model projects concatenated RGB/audio features, applies a
|
|
single-layer GRU, expands the recurrent state into ten anticipation states,
|
|
and uses a shared classifier. The EasyCom configuration uses a 128-step
|
|
window, stride 4, 2048D embedding, 1024D GRU and ten future steps.
|
|
|
|
We reuse the defensible ideas—causal recurrence, shared weights and a ten-step
|
|
anticipation output—but not the implementation. Our GRU is shared across
|
|
anonymous person tracks, reads only crop observations at or before the current
|
|
cutoff, and corrects the already measured pooled Omni baseline. Consequently,
|
|
this method must be described as an EgoSpeak-inspired track-temporal adapter,
|
|
not an EgoSpeak GRU reproduction.
|
|
|
|
## Causal and identity contract
|
|
|
|
- A 5 Hz row with cutoff `(t+1)/5` reads the latest local video frame strictly
|
|
before that cutoff.
|
|
- Track history contains at most `--person-history-frames` real 2 FPS crop
|
|
observations and is ordered oldest to newest.
|
|
- Temporary IDs only link observations. They are never embedded or passed to
|
|
the model, so global ID renaming cannot change a prediction.
|
|
- A source-clip boundary always resets a track. A time gap larger than
|
|
`--person-track-max-gap-seconds` also resets it.
|
|
- Missing, `-1`, `None`, `null` and `unknown` IDs are isolated observations.
|
|
- Duplicate known IDs in one frame fail instead of being silently merged.
|
|
- Empty person sets produce exactly zero visual correction.
|
|
|
|
The existing EasyCom local sidecars already contain crop features, frame times,
|
|
source-clip indices and person IDs; this experiment does not require feature
|
|
re-extraction.
|
|
|
|
## Fair initialization
|
|
|
|
`track-gru-residual` requires `--initialize-pooled-checkpoint`. The loader
|
|
checks dataset, train/dev split, loss, layer fusion, dimensions and local
|
|
feature contract before copying the pooled path. By default those copied
|
|
parameters are frozen. The residual gate starts at zero, so before optimization
|
|
the track model must reproduce the pooled logits exactly, element for element.
|
|
|
|
The matched screening protocol is:
|
|
|
|
- EasyCom Session 4--11: training;
|
|
- Session 12: development/model selection;
|
|
- Session 1--3: final test, run only after history/gap and the method are frozen;
|
|
- 4 epochs, three seeds, identical optimizer/loss for pooled and track runs;
|
|
- 16 observations at 2 FPS (up to 8 seconds), 1-second maximum link gap.
|
|
|
|
This protocol intentionally re-runs the pooled base. Historical no-dev results
|
|
cannot be used as the matched control.
|
|
|
|
## Run
|
|
|
|
After pulling on the L20 machine, run the focused CPU regression tests first:
|
|
|
|
```bash
|
|
python -m unittest \
|
|
test_local_turn_features.LocalTests.test_track_loader_builds_causal_histories_and_resets_tracks \
|
|
test_local_turn_features.LocalTests.test_track_gru_residual_starts_from_exact_pooled_logits \
|
|
test_local_turn_features.RoundTripTests.test_train_infer_easycom_and_ego4d
|
|
```
|
|
|
|
One-seed runtime smoke on an L20:
|
|
|
|
```bash
|
|
EPOCHS=1 \
|
|
SESSIONS="4 5" \
|
|
DEV_SESSIONS=5 \
|
|
SCREEN_SESSIONS=5 \
|
|
CHECKPOINT_ROOT=checkpoints/easycom_track_gru_smoke \
|
|
RESULT_ROOT=results/easycom_track_gru_smoke \
|
|
CUDA_VISIBLE_DEVICES=0 \
|
|
bash run_easycom_track_gru_residual.sh all 0
|
|
```
|
|
|
|
Then use three GPUs in three foreground terminals. These runs report Session
|
|
12 development scores, not the official test set:
|
|
|
|
```bash
|
|
SUMMARIZE=0 CUDA_VISIBLE_DEVICES=0 bash run_easycom_track_gru_residual.sh all 0
|
|
SUMMARIZE=0 CUDA_VISIBLE_DEVICES=1 bash run_easycom_track_gru_residual.sh all 1
|
|
SUMMARIZE=0 CUDA_VISIBLE_DEVICES=2 bash run_easycom_track_gru_residual.sh all 2
|
|
```
|
|
|
|
After all development jobs finish:
|
|
|
|
```bash
|
|
python summarize_local_baselines.py \
|
|
--root results/easycom_track_gru_residual \
|
|
--dataset easycom \
|
|
--protocol causal_valid_only \
|
|
--paired-deltas
|
|
```
|
|
|
|
Only after choosing and freezing the method, evaluate the saved checkpoints
|
|
once on Sessions 1--3:
|
|
|
|
```bash
|
|
CUDA_VISIBLE_DEVICES=0 bash run_easycom_track_gru_residual.sh test 0 1 2
|
|
```
|
|
|
|
This writes and summarizes `results/easycom_track_gru_residual_test` without
|
|
retraining. A single GPU is sufficient because head inference is inexpensive.
|
|
|
|
Paths and hyperparameters can be overridden with the environment variables at
|
|
the top of `run_easycom_track_gru_residual.sh`. The launcher runs in the
|
|
foreground and refuses to overwrite an existing checkpoint or result.
|
|
|
|
## Interpretation and stopping rule
|
|
|
|
The initial screen answers only whether track-temporal visual structure can
|
|
improve the dense EgoSpeak-compatible metric. Advance this branch only if
|
|
same-seed `track_gru - pooled` deltas are directionally consistent on Session
|
|
12 without sacrificing short-horizon wearer AP. The official test is a final
|
|
confirmation, not another model-selection stage.
|
|
Because prior F1 variation was roughly a few tenths of a percentage point,
|
|
tiny single-seed changes are not evidence.
|
|
|
|
If the track branch wins, next run correct-track versus broken-track,
|
|
past-shifted and no-ID temporal controls, followed by person-conditioned audio
|
|
grounding. If it does not win, first compare 8/16/32 observations and optionally
|
|
unfreeze the pooled base using Session 12 only; do not add deeper cross-attention
|
|
merely to rescue a null result.
|
|
|
|
The paper-level claim still requires the frozen competing-risk event evaluator
|
|
and matched-recall false-entry analysis. Dense AP/mAP improvement alone is a
|
|
representation result, not proof that the model understands multi-person floor
|
|
behavior.
|