icassp2027-experiment/docs/easycom_oracle_information.md

# EasyCom causal oracle-person information screen

## Question

Before training a heavier visual grounding model, test whether perfect
participant-resolved **past** speaking activity contains useful information for
future wearer activity beyond the frozen Omni state and recent mixed audio.
This is an information upper-bound experiment, not a deployable input setting.

## Causal contract

- The task grid remains 5 Hz. State row `t` has cutoff `(t+1)/5` seconds.
- Oracle state `t` follows EgoSpeak's released-target conversion exactly:
  source-frame markers `4, 8, 12, ...` are sampled at the corresponding
  `0.2, 0.4, 0.6, ...` second cutoffs, with
  `Start_Frame <= marker < End_Frame`. This is right-edge point sampling, not
  an any-overlap reduction over a 200 ms bin.
- Each person vector stores the last 2 seconds of binary activity followed by
  a visibility/availability mask. Activity is exposed only in bins where that
  participant was visible according to the provided boxes. No future bin,
  future wearer state, transcript text or `Target_of_Speech` value is input.
- Only non-wearer participant IDs present in the latest provided-box frame
  strictly before the cutoff receive slots. IDs route annotations during
  preparation but are not exposed as embeddings to the head.
- Transcript-derived wearer/others activity must match `target_perframe`
  exactly. Any mismatch fails closed; there is no tolerance flag.
- Oracle sidecars use `easycom-oracle-person-history-v1` and are accepted only
  by the explicit `oracle-frame-set-attention` mode. Normal local-feature
  loading rejects them because they use speech labels.

## Conditions

| Condition | Information exposed | Purpose |
|---|---|---|
| `audio` | Omni `G` plus the existing recent local audio `A` | deployable baseline |
| `visibility-only` | current visible-person set, but all activity bits are zero | parameter/person-count control |
| `aggregate-visible` | one OR-pooled history over currently visible people | aggregate floor-information upper bound |
| `per-person` | one correct activity history per currently visible participant | main oracle information test |
| `shuffled-id` | preserves the number active at every lag but independently reassigns activity between people | destroys identity continuity |
| `delayed` | correct per-person histories delayed by 1 second | destroys current temporal alignment without future leakage |

The decisive comparison is `per-person` against `aggregate-visible`,
`shuffled-id` and `delayed`, not only against `audio`. A gain over `audio` alone
could come from perfect aggregate VAD rather than person continuity.

## Remote preparation and one-seed Session-12 screen

The source local sidecars provide only mixed audio and the visibility/box
timeline. Their learned visual vectors are not copied into the oracle input.

```bash
cd /data1/yfzx/wangx/Code/icassp2027-experiment

export DATASET_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom
export FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
export TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
export LOCAL_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/local_qwen_1s_encoded_tf4576
export ORACLE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/oracle_person_history_2s
export CHECKPOINT_ROOT=checkpoints/easycom_oracle_information_4ep
export RESULT_ROOT=results/easycom_oracle_information_4ep

CUDA_VISIBLE_DEVICES=0 bash run_easycom_oracle_information.sh all 0
```

`all` prepares the five oracle conditions, trains on Sessions 4--11, selects
the checkpoint on Session 12, runs Session-12 inference and prints the standard
dense AP/mAP screen. Existing compatible oracle sidecars are validated and
reused, so interrupted preparation can resume. It deliberately does not run
Sessions 1--3.

If preparation reports any transcript/target mismatch, stop and inspect it;
the preparation command has no mismatch-tolerance option.

## Locked four-epoch train-all protocol

The final protocol fixes the training budget at four epochs, trains on all
official EasyCom training sessions (Sessions 4--12), performs no development
checkpoint selection, and evaluates Sessions 1--3 once:

```bash
CUDA_VISIBLE_DEVICES=0 bash run_easycom_oracle_information.sh final 0 1 2
```

The `final` stage enforces `EPOCHS=4` and exactly Sessions 4--12, clears the
development-session list internally, and saves the last epoch. Its artifacts
use separate `_train4_12_fixed4` checkpoint/result roots by default, so they
cannot overwrite the historical Session-12 development screen. Override those
roots only with `FINAL_CHECKPOINT_ROOT` and `FINAL_RESULT_ROOT`.

mAP itself does not require threshold calibration. Nevertheless, the official
test results must not be used to choose a variant, epoch, loss, fusion design,
or any later hyperparameter. Run `final` only after those decisions are frozen;
repeatedly inspecting Sessions 1--3 and modifying the method would contaminate
the official test split.

## Follow-up

To reproduce the historical Session-12 development screen for seeds 1 and 2,
use the already prepared sidecars:

```bash
CUDA_VISIBLE_DEVICES=0 bash run_easycom_oracle_information.sh screen 1 2
```

The older two-stage equivalent of final test inference is retained for
reproducibility, but the locked protocol above is preferred for new final runs:

```bash
VARIANTS="audio aggregate-visible per-person shuffled-id delayed" \
CUDA_VISIBLE_DEVICES=0 \
bash run_easycom_oracle_information.sh test 0 1 2
```

## Interpretation

- `per-person > aggregate-visible` and `per-person > shuffled-id`: speaker
  continuity has information value; proceed to learned causal ASD/tracks.
- `aggregate-visible > audio`, but no per-person advantage: perfect aggregate
  activity helps, but the current target does not justify an identity claim.
- oracle conditions do not improve: do not spend the main GPU budget on a more
  complex person encoder until the task/head/evaluator is reconsidered.
- oracle improves but learned Qwen face features do not: this supports a
  speaker-grounding bottleneck and motivates EgoASD-like AV evidence.

Dense 10-horizon mAP is only a screening signal. The publication claim still
requires the frozen competing-risk-censored entry/yield evaluator and matched
wearer-onset-recall false-entry analysis.