icassp2027-experiment/results/easycom_transition_auxiliary_4ep_2026-08-27/README.md

# EasyCom transition-aware auxiliary-head ablation

Recorded on 2026-08-27 from user-reported remote evaluation output. This is a
preliminary two-seed snapshot: seed 2 has not been reported, so these numbers
must not yet be used as a final multi-seed paper result.

## Experiment contract

- Dataset: EasyCom.
- Train split: Sessions 4 through 12, with no development set.
- Test split: EgoSpeak official Sessions 3, 2, and 1.
- Backbone: frozen Qwen2.5-Omni-3B Thinker.
- Representation: 8-second open-stream replay, 2 FPS video, 5 Hz states, final
  four query layers stacked.
- Readout: concat-MLP followed by the activity MLP.
- Activity objective: LSTR-MCE.
- Training budget: 4 epochs, batch size 512, learning rate 7e-5, weight decay
  5e-5, dropout 0.2, 1.6 warmup epochs, warmup factor 0.3.
- Completed seeds: 0 and 1. Missing seed: 2.
- Primary evaluation protocol: causal-valid-only activity AP/mAP.

A positive transition at future horizon h means that the complete three-label
activity state changes between target rows t+h-1 and t+h.

- R1: activity loss + 0.2 times weighted transition BCE, with positive weight
  3.
- R2: R1 loss - 0.1 times exact expected weighted transition correctness.

## Individual results

All values below are percentage points.

| Variant | Seed | 0.2s mAP | BG AP | Wearer AP | Others AP | Mean mAP | Transition 0.2s AP | Transition 0.2s F1@0.5 | Transition mean AP |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| R1 | 0 | 73.51590 | 56.84019 | 77.92448 | 85.78303 | 60.86360 | 15.19153 | 0.46172 | 13.04354 |
| R1 | 1 | 73.78692 | 56.92102 | 78.63090 | 85.80884 | 60.82023 | 15.32492 | 1.14068 | 13.06174 |
| R2 | 0 | 73.53288 | 56.89701 | 77.92732 | 85.77432 | 60.86528 | 15.37608 | 0.00000 | 13.11854 |
| R2 | 1 | 73.82947 | 56.97474 | 78.69340 | 85.82027 | 60.84178 | 15.47356 | 0.84227 | 13.13820 |

## Two-seed mean and sample standard deviation

| Variant | 0.2s mAP | Wearer AP | Others AP | Mean mAP | Transition 0.2s AP | Transition mean AP |
|---|---:|---:|---:|---:|---:|---:|
| R1 | 73.65141 +/- 0.19164 | 78.27769 +/- 0.49951 | 85.79594 +/- 0.01825 | 60.84192 +/- 0.03067 | 15.25823 +/- 0.09432 | 13.05264 +/- 0.01287 |
| R2 | 73.68118 +/- 0.20972 | 78.31036 +/- 0.54169 | 85.79729 +/- 0.03249 | 60.85353 +/- 0.01662 | 15.42482 +/- 0.06893 | 13.12837 +/- 0.01390 |

## Matched-seed comparison with LSTR-MCE

The baseline values are taken from the existing four-epoch loss-structure
ablation. Using only the same completed seeds 0 and 1 avoids comparing a
two-seed transition result with a three-seed baseline mean.

| Comparison | Delta 0.2s mAP | Delta BG AP | Delta wearer AP | Delta others AP | Delta mean mAP |
|---|---:|---:|---:|---:|---:|
| R1 minus LSTR-MCE | +0.04975 | +0.02633 | +0.05942 | +0.06350 | +0.04949 |
| R2 minus LSTR-MCE | +0.07951 | +0.08160 | +0.09209 | +0.06486 | +0.06110 |
| R2 minus R1 | +0.02977 | +0.05527 | +0.03267 | +0.00136 | +0.01162 |

All activity changes are much smaller than the observed seed variation. The
current evidence therefore does not establish a meaningful activity
anticipation improvement.

## Transition diagnostic

The 0.2-second transition prevalence is 11.21718%, which is also the expected
AP of a random ranking. R1 reaches 15.19--15.32% AP, or 1.35--1.37 times the
random baseline. R2 reaches 15.38--15.47%, or 1.37--1.38 times random. The
auxiliary head therefore learns weak but non-random transition ranking.

F1 at threshold 0.5 is nearly zero because recall is only 0--0.58%. This fixed
threshold is not a reliable model-quality measure under weighted BCE: positive
weight 3 changes probability calibration and remains below the approximately
7.9 negative-to-positive ratio implied by the test prevalence. Threshold-free
AP is the appropriate diagnostic here. Any deployable threshold must be
selected on training-held-out data, not the test set.

R2 improves transition AP over R1 by only 0.16660 points at 0.2 seconds and
0.07573 points on the ten-horizon mean. Its activity mean-mAP improvement over
R1 is only 0.01162 points. Exact expected correctness supplies almost the same
gradient direction as weighted BCE, so this result is consistent with the two
terms being largely redundant rather than R2 learning a new sequential policy.

## Current interpretation

This is a preliminary null ablation:

> Exact 200-ms transition supervision provides weak non-random boundary
> ranking but no meaningful improvement in the primary activity-anticipation
> metric. Adding exact expected correctness produces negligible further gain.

This experiment should not be described as interactive reinforcement learning.
R2 computes the binary-action expectation analytically on fixed offline labels;
it is a low-variance supervised reward control.

## Required follow-up

1. Complete and report seed 2 before freezing the table.
2. Do not select a transition threshold or tune positive/reward weights on the
   test sessions.
3. If the transition line is continued, replace the sparse exact-boundary
   target with a pre-specified cumulative transition-hazard target: whether at
   least one transition occurs within the next h steps.
4. If that structured target also fails to improve activity mAP, treat
   transition auxiliary learning on frozen states as a rejected direction and
   move experiment budget to representation or temporal modeling.

## Provenance

- Feature root:
  /data1/yfzx/wangx/Datasets/easycom/omni_features/open_stream_2fps_8s_last4_stack
- Target root:
  /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame
- Remote result roots:
  results/easycom-transition-r1-4ep-seed-N and
  results/easycom-transition-r2-4ep-seed-N
- Source: user-reported summary and diagnostic console output.
- Raw remote metrics JSON archived locally: no.
- Full per-horizon activity and transition curves archived locally: no.

Machine-readable records are in summary.json and metrics.csv. The received
console values are preserved in raw_console.txt.