icassp2027-experiment/docs/easycom_oracle_information.md

EasyCom causal oracle-person information screen

Question

Before training a heavier visual grounding model, test whether perfect participant-resolved past speaking activity contains useful information for future wearer activity beyond the frozen Omni state and recent mixed audio. This is an information upper-bound experiment, not a deployable input setting.

Causal contract

• The task grid remains 5 Hz. State row t has cutoff (t+1)/5 seconds.
• Oracle state t follows EgoSpeak's released-target conversion exactly: source-frame markers 4, 8, 12, ... are sampled at the corresponding 0.2, 0.4, 0.6, ... second cutoffs, with Start_Frame <= marker < End_Frame. This is right-edge point sampling, not an any-overlap reduction over a 200 ms bin.
• Each original person vector stores the last 2 seconds of binary activity followed by a visibility/availability mask. Activity is exposed only in bins where that participant was visible according to the provided boxes. No future bin, future wearer state or transcript text is input. The separately named addressee diagnostic below additionally uses privileged Target_of_Speech; it must not be described as an online/deployable feature.
• Only non-wearer participant IDs present in the latest provided-box frame strictly before the cutoff receive slots. IDs route annotations during preparation but are not exposed as embeddings to the head.
• Transcript-derived wearer/others activity must match target_perframe exactly. Any mismatch fails closed; there is no tolerance flag.
• Oracle sidecars use easycom-oracle-person-history-v1 and are accepted only by the explicit oracle-frame-set-attention mode. Normal local-feature loading rejects them because they use speech labels.

Conditions

Condition	Information exposed	Purpose
audio	Omni G plus the existing recent local audio A	deployable baseline
visibility-only	current visible-person set, but all activity bits are zero	parameter/person-count control
aggregate-visible	one OR-pooled history over currently visible people	aggregate floor-information upper bound
aggregate-visible-address-control	aggregate-visible plus an all-zero history channel	capacity control for addressee input
aggregate-visible-addressee	aggregate-visible plus past direct-to-wearer annotation bins	analysis-only addressee information upper bound
per-person	one correct activity history per currently visible participant	main oracle information test
shuffled-id	preserves the number active at every lag but independently reassigns activity between people	destroys identity continuity
delayed	correct per-person histories delayed by 1 second	destroys current temporal alignment without future leakage

The decisive comparison is per-person against aggregate-visible, shuffled-id and delayed, not only against audio. A gain over audio alone could come from perfect aggregate VAD rather than person continuity.

For the separate addressee question, the decisive comparison is aggregate-visible-addressee against both aggregate-visible and aggregate-visible-address-control. Group/None/unknown targets are audited and not treated as trustworthy semantic negatives. The full diagnostic commands are in easycom_floor_event_diagnostics.md.

Remote preparation and one-seed Session-12 screen

The source local sidecars provide only mixed audio and the visibility/box timeline. Their learned visual vectors are not copied into the oracle input.

cd /data1/yfzx/wangx/Code/icassp2027-experiment

export DATASET_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom
export FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
export TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
export LOCAL_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/local_qwen_1s_encoded_tf4576
export ORACLE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/oracle_person_history_2s
export CHECKPOINT_ROOT=checkpoints/easycom_oracle_information_4ep
export RESULT_ROOT=results/easycom_oracle_information_4ep

CUDA_VISIBLE_DEVICES=0 bash run_easycom_oracle_information.sh all 0

all prepares the five oracle conditions, trains on Sessions 4--11, selects the checkpoint on Session 12, runs Session-12 inference and prints the standard dense AP/mAP screen. Existing compatible oracle sidecars are validated and reused, so interrupted preparation can resume. It deliberately does not run Sessions 1--3.

If preparation reports any transcript/target mismatch, stop and inspect it; the preparation command has no mismatch-tolerance option.

Locked four-epoch train-all protocol

The final protocol fixes the training budget at four epochs, trains on all official EasyCom training sessions (Sessions 4--12), performs no development checkpoint selection, and evaluates Sessions 1--3 once:

CUDA_VISIBLE_DEVICES=0 bash run_easycom_oracle_information.sh final 0 1 2

The final stage overrides any inherited EPOCHS value with 4, requires exactly Sessions 4--12, clears the development-session list internally, and saves the last epoch. It validates this configuration before preparing oracle sidecars. Its artifacts use separate _train4_12_fixed4 checkpoint/result roots by default, so they cannot overwrite the historical Session-12 development screen. Override those roots only with FINAL_CHECKPOINT_ROOT and FINAL_RESULT_ROOT.

mAP itself does not require threshold calibration. Nevertheless, the official test results must not be used to choose a variant, epoch, loss, fusion design, or any later hyperparameter. Run final only after those decisions are frozen; repeatedly inspecting Sessions 1--3 and modifying the method would contaminate the official test split.

Follow-up

To reproduce the historical Session-12 development screen for seeds 1 and 2, use the already prepared sidecars:

CUDA_VISIBLE_DEVICES=0 bash run_easycom_oracle_information.sh screen 1 2

The older two-stage equivalent of final test inference is retained for reproducibility, but the locked protocol above is preferred for new final runs:

VARIANTS="audio aggregate-visible per-person shuffled-id delayed" \
CUDA_VISIBLE_DEVICES=0 \
bash run_easycom_oracle_information.sh test 0 1 2

Interpretation

• per-person > aggregate-visible and per-person > shuffled-id: speaker continuity has information value; proceed to learned causal ASD/tracks.
• aggregate-visible > audio, but no per-person advantage: perfect aggregate activity helps, but the current target does not justify an identity claim.
• oracle conditions do not improve: do not spend the main GPU budget on a more complex person encoder until the task/head/evaluator is reconsidered.
• oracle improves but learned Qwen face features do not: this supports a speaker-grounding bottleneck and motivates EgoASD-like AV evidence.

Dense 10-horizon mAP is only a screening signal. The publication claim still requires the frozen competing-risk-censored entry/yield evaluator and matched wearer-onset-recall false-entry analysis.