6.2 KiB
EasyCom floor-event diagnosis: timing, objective, and addressee
These three experiments explain why a dense future-activity oracle can retain reasonable mAP while response/yield event AP remains low. They are diagnostics, not three proposed paper contributions. Keep the primary paper protocol as the pre-boundary competing-risk evaluation until the evidence says otherwise.
What each experiment isolates
| Diagnostic | Only intended change | If it improves clearly |
|---|---|---|
| pre-boundary vs post-boundary | whether the aggregate-other boundary has already been observed at the decision source | much of the error is boundary anticipation, not weak AV encoding |
| dense activity head vs direct event head | future-state loss vs response/yield loss, using the same frozen inputs | the dense objective/ranking conversion is the main mismatch |
| addressee vs zero-channel control | explicit past Target_of_Speech == wearer_id history |
aggregate activity misses conversational routing information |
Do not compare raw AP across the pre- and post-boundary protocols as if they were the same task. Compare methods within one timing protocol, and use the pre/post difference only as a difficulty decomposition.
Shared remote environment
cd /data1/yfzx/wangx/Code/icassp2027-experiment
export DATASET_ROOT=/data1/yfzx/wangx/Datasets/easycom/extracted_easycom
export FEATURE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
export TARGET_ROOT=/data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
export LOCAL_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/local_qwen_1s_encoded_tf4576
export ORACLE_ROOT=/data1/yfzx/wangx/Datasets/easycom/omni_features/oracle_person_history_2s
Change only paths that differ on the remote host. The default feature mode is
open-stream-replay.
1. Post-boundary reactive diagnostic for the three existing runs
Point the wrapper at already aligned inference directories. Each directory
must contain pred/Session_*.npy and valid/Session_*.npy.
export POOLED_AV_RUN=/path/to/pooled_av_result
export ORACLE_RUN=/path/to/aggregate_visible_oracle_result
export MINIROAD_RUN=/path/to/converted_miniroada_result
bash run_easycom_post_boundary_eval.sh
Without CALIBRATION_SESSIONS, the wrapper explicitly marks threshold results
as exploratory and selects them in-sample. AP is still useful for this first
diagnosis. For a confirmatory result, provide training-side predictions in the
same run directories and use, for example:
export CALIBRATION_SESSIONS="Session_12"
bash run_easycom_post_boundary_eval.sh
The JSON protocol ID becomes
wearer_floor_event_post_boundary_diagnostic_v1. Simultaneous wearer changes
at the observed boundary and zero-length windows before an immediate competing
boundary are excluded and counted explicitly.
2. Direct response/yield event head
The direct probe uses frozen Omni states, the same mixed-audio vectors and the
same aggregate-visible past-activity oracle. Its layer fusion, audio
projection, oracle person attention and shared MLP trunk are the same
TurnPredictionHead components as the dense oracle; only the final two-event
readout and event loss differ. It trains on Sessions 4--11, selects epoch and
thresholds on Session 12, and initially evaluates only Session 12. First run a
one-epoch Session-11/12 shape, mask and checkpoint smoke test:
CUDA_VISIBLE_DEVICES=0 bash run_easycom_floor_event_probes.sh smoke
Then run the single aggregate oracle diagnosis:
CUDA_VISIBLE_DEVICES=0 \
VARIANTS="aggregate-visible" \
bash run_easycom_floor_event_probes.sh all 0
This is the clean comparison against the existing dense aggregate-visible oracle head. A large direct-head gain supports objective mismatch; a small gain means the oracle information itself is probably weak for the event decision.
3. Addressee oracle with a capacity-matched control
Run all three direct-probe variants:
CUDA_VISIBLE_DEVICES=0 bash run_easycom_floor_event_probes.sh all 0
The variants are:
aggregate-visible: 2 s aggregate activity plus observation mask;aggregate-visible-address-control: the same input plus an all-zero channel;aggregate-visible-addressee: the same input plus past bins in which a visible non-wearer was explicitly annotated as addressing wearer ID 2.
Group, None, unknown and other-person targets are not coerced into reliable
negative labels. The addressee channel is privileged, annotation-derived and
analysis-only; it is not an online feature or a deployable causal input.
The runner writes one summary.json and metrics.csv per variant and a compact
comparison_dev.csv. Inspect that comparison before touching Sessions 1--3.
Optional frozen test evaluation
Only after the variants, epochs and interpretation rule are fixed from Session 12, apply the saved probes to official test Sessions 1--3:
CUDA_VISIBLE_DEVICES=0 bash run_easycom_floor_event_probes.sh test 0
This reuses the saved Session-12 thresholds and writes comparison_test.csv.
Do not repeatedly inspect test results and redesign the probe.
Minimum interpretation rule
For response, jointly inspect mean AP, 2 s AP, achieved response recall, false entry rate, and short-resume trigger rate. For yield, inspect mean AP, 2 s AP, achieved yield recall, and false-stop rate. Denominators and censoring counts must match before comparing variants.
The three outcomes lead to different paper decisions:
- Only post-boundary improves: retain the main anticipation task and frame the difficulty as predicting a boundary before it is observable. Do not claim that a larger event head solves grounding.
- Direct event supervision improves: the main lesson is target alignment; build the method around event-aware adaptation and keep dense mAP secondary.
- Addressee improves over both aggregate and zero control: conversational routing is the missing information. This motivates a deployable addressee or addressedness estimator, but the oracle itself is not the method.
- None improves: the current timing-derived event definition or available observations are insufficient. Revisit annotation/task definition before a heavier fusion architecture.