72 lines
3.8 KiB
Markdown
72 lines
3.8 KiB
Markdown
# Stage-one frozen EgoSpeak + Omni residual screen
|
|
|
|
## Provenance and scope
|
|
|
|
These values were reported by the user on 2026-09-16, not recomputed on this
|
|
host. The preceding launcher command specifies seed 0, four fixed epochs on
|
|
EasyCom Sessions 4--12, and final evaluation on Sessions 1--3. The reported
|
|
run is `results/easycom_egospeak_omni_residual_seed0/checkpoint_last.pt`; its
|
|
actual checkpoint hash/config and sample counts were not supplied and remain
|
|
unverified. The base is the released `miniroad_easycom_AV.pth` checkpoint.
|
|
Implementation: [`stage-one runbook`](../../docs/easycom_egospeak_omni_residual.md).
|
|
|
|
The original MiniROADA A+V path and Omni backbone remain frozen; only the
|
|
tail-phase-aware residual fusion is trained. Existing Omni states use the
|
|
8-s open-stream configuration and a fixed mean over the last-four stack.
|
|
Both full-timeline and causal-valid dense numbers are retained in
|
|
[`metrics.csv`](metrics.csv). All numbers in this record are percentages;
|
|
deltas are percentage points. Displayed values are rounded console output.
|
|
|
|
## Findings
|
|
|
|
| Metric | Frozen EgoSpeak | Fused | Delta |
|
|
|---|---:|---:|---:|
|
|
| Causal-valid 0.2-s dense mAP | 74.204 | 77.739 | +3.535 |
|
|
| Causal-valid 0.2-s wearer AP | 63.750 | 72.716 | +8.966 |
|
|
| Causal-valid ten-horizon dense mean mAP | 61.193 | 62.714 | +1.521 |
|
|
| Full-timeline ten-horizon dense mean mAP | 61.155 | 62.676 | +1.521 |
|
|
| Ten-horizon raw onset endpoint AP | 24.491 | 25.091 | +0.600 |
|
|
| Ten-horizon raw offset endpoint AP | 38.024 | 40.604 | +2.580 |
|
|
|
|
Offset AP improves at all ten horizons, by about 2.8--3.5 points at
|
|
0.4--1.2 s. Onset AP improves at nine horizons; its largest gain is +1.634
|
|
points at 0.8 s, while long-horizon differences are small. This is a
|
|
single-seed positive screen, strongest for future wearer cessation. It is
|
|
not a significance test or evidence of semantic floor-yield understanding.
|
|
A parameter-matched EgoSpeak-only residual control and additional seeds
|
|
remain necessary to attribute the gain specifically to Omni.
|
|
|
|
## Raw anticipation definition
|
|
|
|
For each horizon h=0.2,...,2.0 s, use valid source/target pairs from the same
|
|
session. Onset is evaluated only when wearer activity W(t)=0: positive if
|
|
W(t+h)=1, negative if W(t+h)=0, with score p_wearer(t,h). Offset is evaluated
|
|
only when W(t)=1: positive if W(t+h)=0, negative if W(t+h)=1, with score
|
|
1-p_wearer(t,h). AP ranks all eligible samples pooled over Sessions 1--3;
|
|
the headline averages ten horizon APs. This is endpoint state-change
|
|
anticipation, not one-event-one-vote timestamp detection. Intermediate
|
|
transitions can be invisible when source and endpoint states agree.
|
|
|
|
The evaluator also computed exploratory/in-sample operating points, but
|
|
only threshold-free endpoint APs are recorded here. They need no Session-4
|
|
threshold calibration. Do not use the accompanying in-sample rate columns
|
|
as confirmatory test results.
|
|
|
|
## Annotation decision and next stage
|
|
|
|
The user suspects issues with the provisional LLM-assisted semantic
|
|
annotations. Opportunity-conditioned ENTRY/YIELD evaluation is therefore
|
|
temporarily paused pending an annotation audit. Historical provisional
|
|
metrics remain archived; they are neither human-reviewed ground truth nor
|
|
discarded because of model performance. No new semantic result or semantic
|
|
loss is introduced in this screen.
|
|
|
|
The next requested implementation is selective MiniROADA GRU/anticipation
|
|
fine-tuning from the stage-one checkpoint, keeping original feature
|
|
extractors and Omni frozen. This is an exploratory continuation of the
|
|
previously documented route, not passage of the still-unfinished P0 semantic
|
|
evaluation gate. Do not tune windows, loss weights or stopping on Sessions
|
|
1--3; compare fixed budgets and preserve the frozen stage-one/original base
|
|
as references. A four-epoch continuation has eight total adaptation epochs
|
|
and is not a matched-budget comparison against the four-epoch first stage.
|