EasyCom floor-event diagnosis: timing, objective, and addressee

These three experiments explain why a dense future-activity oracle can retain reasonable mAP while response/yield event AP remains low. They are diagnostics, not three proposed paper contributions. Keep the primary paper protocol as the pre-boundary competing-risk evaluation until the evidence says otherwise.

What each experiment isolates

Diagnostic	Only intended change	If it improves clearly
pre-boundary vs post-boundary	whether the aggregate-other boundary has already been observed at the decision source	much of the error is boundary anticipation, not weak AV encoding
dense activity head vs direct event head	future-state loss vs response/yield loss, using the same frozen inputs	the dense objective/ranking conversion is the main mismatch
addressee vs zero-channel control	explicit past `Target_of_Speech == wearer_id` history	aggregate activity misses conversational routing information

Do not compare raw AP across the pre- and post-boundary protocols as if they were the same task. Compare methods within one timing protocol, and use the pre/post difference only as a difficulty decomposition.

Shared remote environment

cd /data1/yfzx/wangx/Code/icassp2027-experiment

export DATASET_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom
export FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
export TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
export LOCAL_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/local_qwen_1s_encoded_tf4576
export ORACLE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/oracle_person_history_2s

Change only paths that differ on the remote host. The default feature mode is open-stream-replay.

1. Post-boundary reactive diagnostic for the three existing runs

Point the wrapper at already aligned inference directories. Each directory must contain pred/Session_*.npy and valid/Session_*.npy.

export POOLED_AV_RUN=/path/to/pooled_av_result
export ORACLE_RUN=/path/to/aggregate_visible_oracle_result
export MINIROAD_RUN=/path/to/converted_miniroada_result

bash run_easycom_post_boundary_eval.sh

Without CALIBRATION_SESSIONS, the wrapper explicitly marks threshold results as exploratory and selects them in-sample. AP is still useful for this first diagnosis. For a confirmatory result, provide training-side predictions in the same run directories and use, for example:

export CALIBRATION_SESSIONS="Session_12"
bash run_easycom_post_boundary_eval.sh

The JSON protocol ID becomes wearer_floor_event_post_boundary_diagnostic_v1. Simultaneous wearer changes at the observed boundary and zero-length windows before an immediate competing boundary are excluded and counted explicitly.

2. Direct response/yield event head

The direct probe uses frozen Omni states, the same mixed-audio vectors and the same aggregate-visible past-activity oracle. Its layer fusion, audio projection, oracle person attention and shared MLP trunk are the same TurnPredictionHead components as the dense oracle; only the final two-event readout and event loss differ. It trains on Sessions 4--11, selects epoch and thresholds on Session 12, and initially evaluates only Session 12. First run a one-epoch Session-11/12 shape, mask and checkpoint smoke test:

CUDA_VISIBLE_DEVICES=0 bash run_easycom_floor_event_probes.sh smoke

Then run the single aggregate oracle diagnosis:

CUDA_VISIBLE_DEVICES=0 \
VARIANTS="aggregate-visible" \
bash run_easycom_floor_event_probes.sh all 0

This is the clean comparison against the existing dense aggregate-visible oracle head. A large direct-head gain supports objective mismatch; a small gain means the oracle information itself is probably weak for the event decision.

3. Addressee oracle with a capacity-matched control

Run all three direct-probe variants:

CUDA_VISIBLE_DEVICES=0 bash run_easycom_floor_event_probes.sh all 0

The variants are:

aggregate-visible: 2 s aggregate activity plus observation mask;
aggregate-visible-address-control: the same input plus an all-zero channel;
aggregate-visible-addressee: the same input plus past bins in which a visible non-wearer was explicitly annotated as addressing wearer ID 2.

Group, None, unknown and other-person targets are not coerced into reliable negative labels. The addressee channel is privileged, annotation-derived and analysis-only; it is not an online feature or a deployable causal input.

The runner writes one summary.json and metrics.csv per variant and a compact comparison_dev.csv. Inspect that comparison before touching Sessions 1--3.

Optional frozen test evaluation

Only after the variants, epochs and interpretation rule are fixed from Session 12, apply the saved probes to official test Sessions 1--3:

CUDA_VISIBLE_DEVICES=0 bash run_easycom_floor_event_probes.sh test 0

This reuses the saved Session-12 thresholds and writes comparison_test.csv. Do not repeatedly inspect test results and redesign the probe.

Minimum interpretation rule

For response, jointly inspect mean AP, 2 s AP, achieved response recall, false entry rate, and short-resume trigger rate. For yield, inspect mean AP, 2 s AP, achieved yield recall, and false-stop rate. Denominators and censoring counts must match before comparing variants.

The three outcomes lead to different paper decisions:

Only post-boundary improves: retain the main anticipation task and frame the difficulty as predicting a boundary before it is observable. Do not claim that a larger event head solves grounding.
Direct event supervision improves: the main lesson is target alignment; build the method around event-aware adaptation and keep dense mAP secondary.
Addressee improves over both aggregate and zero control: conversational routing is the missing information. This motivates a deployable addressee or addressedness estimator, but the oracle itself is not the method.
None improves: the current timing-derived event definition or available observations are insufficient. Revisit annotation/task definition before a heavier fusion architecture.

6.2 KiB Raw Blame History