icassp2027-experiment/docs/easycom_oracle_information.md

# EasyCom causal oracle-person information screen

## Question

Before training a heavier visual grounding model, test whether perfect
participant-resolved **past** speaking activity contains useful information for
future wearer activity beyond the frozen Omni state and recent mixed audio.
This is an information upper-bound experiment, not a deployable input setting.

## Causal contract

- The task grid remains 5 Hz. State row `t` has cutoff `(t+1)/5` seconds.
- Oracle state `t` follows EgoSpeak's released-target conversion exactly:
  source-frame markers `4, 8, 12, ...` are sampled at the corresponding
  `0.2, 0.4, 0.6, ...` second cutoffs, with
  `Start_Frame <= marker < End_Frame`. This is right-edge point sampling, not
  an any-overlap reduction over a 200 ms bin.
- Each original person vector stores the last 2 seconds of binary activity
  followed by a visibility/availability mask. Activity is exposed only in bins
  where that participant was visible according to the provided boxes. No future
  bin, future wearer state or transcript text is input. The separately named
  addressee diagnostic below additionally uses privileged `Target_of_Speech`;
  it must not be described as an online/deployable feature.
- Only non-wearer participant IDs present in the latest provided-box frame
  strictly before the cutoff receive slots. IDs route annotations during
  preparation but are not exposed as embeddings to the head.
- Transcript-derived wearer/others activity must match `target_perframe`
  exactly. Any mismatch fails closed; there is no tolerance flag.
- Oracle sidecars use `easycom-oracle-person-history-v1` and are accepted only
  by the explicit `oracle-frame-set-attention` mode. Normal local-feature
  loading rejects them because they use speech labels.

## Conditions

| Condition | Information exposed | Purpose |
|---|---|---|
| `audio` | Omni `G` plus the existing recent local audio `A` | deployable baseline |
| `visibility-only` | current visible-person set, but all activity bits are zero | parameter/person-count control |
| `aggregate-visible` | one OR-pooled history over currently visible people | aggregate floor-information upper bound |
| `aggregate-visible-address-control` | aggregate-visible plus an all-zero history channel | capacity control for addressee input |
| `aggregate-visible-addressee` | aggregate-visible plus past direct-to-wearer annotation bins | analysis-only addressee information upper bound |
| `per-person` | one correct activity history per currently visible participant | main oracle information test |
| `shuffled-id` | preserves the number active at every lag but independently reassigns activity between people | destroys identity continuity |
| `delayed` | correct per-person histories delayed by 1 second | destroys current temporal alignment without future leakage |

The decisive comparison is `per-person` against `aggregate-visible`,
`shuffled-id` and `delayed`, not only against `audio`. A gain over `audio` alone
could come from perfect aggregate VAD rather than person continuity.

For the separate addressee question, the decisive comparison is
`aggregate-visible-addressee` against both `aggregate-visible` and
`aggregate-visible-address-control`. Group/None/unknown targets are audited and
not treated as trustworthy semantic negatives. The full diagnostic commands
are in [`easycom_floor_event_diagnostics.md`](easycom_floor_event_diagnostics.md).

## Remote preparation and one-seed Session-12 screen

The source local sidecars provide only mixed audio and the visibility/box
timeline. Their learned visual vectors are not copied into the oracle input.

```bash
cd /data1/yfzx/wangx/Code/icassp2027-experiment

export DATASET_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom
export FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
export TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
export LOCAL_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/local_qwen_1s_encoded_tf4576
export ORACLE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/oracle_person_history_2s
export CHECKPOINT_ROOT=checkpoints/easycom_oracle_information_4ep
export RESULT_ROOT=results/easycom_oracle_information_4ep

CUDA_VISIBLE_DEVICES=0 bash run_easycom_oracle_information.sh all 0
```

`all` prepares the five oracle conditions, trains on Sessions 4--11, selects
the checkpoint on Session 12, runs Session-12 inference and prints the standard
dense AP/mAP screen. Existing compatible oracle sidecars are validated and
reused, so interrupted preparation can resume. It deliberately does not run
Sessions 1--3.

If preparation reports any transcript/target mismatch, stop and inspect it;
the preparation command has no mismatch-tolerance option.

## Locked four-epoch train-all protocol

The final protocol fixes the training budget at four epochs, trains on all
official EasyCom training sessions (Sessions 4--12), performs no development
checkpoint selection, and evaluates Sessions 1--3 once:

```bash
CUDA_VISIBLE_DEVICES=0 bash run_easycom_oracle_information.sh final 0 1 2
```

The `final` stage overrides any inherited `EPOCHS` value with `4`, requires
exactly Sessions 4--12, clears the development-session list internally, and
saves the last epoch. It validates this configuration before preparing oracle
sidecars. Its artifacts
use separate `_train4_12_fixed4` checkpoint/result roots by default, so they
cannot overwrite the historical Session-12 development screen. Override those
roots only with `FINAL_CHECKPOINT_ROOT` and `FINAL_RESULT_ROOT`.

mAP itself does not require threshold calibration. Nevertheless, the official
test results must not be used to choose a variant, epoch, loss, fusion design,
or any later hyperparameter. Run `final` only after those decisions are frozen;
repeatedly inspecting Sessions 1--3 and modifying the method would contaminate
the official test split.

## Follow-up

To reproduce the historical Session-12 development screen for seeds 1 and 2,
use the already prepared sidecars:

```bash
CUDA_VISIBLE_DEVICES=0 bash run_easycom_oracle_information.sh screen 1 2
```

The older two-stage equivalent of final test inference is retained for
reproducibility, but the locked protocol above is preferred for new final runs:

```bash
VARIANTS="audio aggregate-visible per-person shuffled-id delayed" \
CUDA_VISIBLE_DEVICES=0 \
bash run_easycom_oracle_information.sh test 0 1 2
```

## Interpretation

- `per-person > aggregate-visible` and `per-person > shuffled-id`: speaker
  continuity has information value; proceed to learned causal ASD/tracks.
- `aggregate-visible > audio`, but no per-person advantage: perfect aggregate
  activity helps, but the current target does not justify an identity claim.
- oracle conditions do not improve: do not spend the main GPU budget on a more
  complex person encoder until the task/head/evaluator is reconsidered.
- oracle improves but learned Qwen face features do not: this supports a
  speaker-grounding bottleneck and motivates EgoASD-like AV evidence.

Dense 10-horizon mAP is only a screening signal. The publication claim still
requires the frozen competing-risk-censored entry/yield evaluator and matched
wearer-onset-recall false-entry analysis.