diff --git a/PROJECT_CONTEXT.md b/PROJECT_CONTEXT.md index e757df3..6c2a7a9 100644 --- a/PROJECT_CONTEXT.md +++ b/PROJECT_CONTEXT.md @@ -135,7 +135,16 @@ result. described as upstream Stage-3 code because that entry point remains unpublished. See the [`runbook`](docs/easycom_muvap_stage3_baseline.md). -- `egospeak-omni-zero-residual` is implemented but unmeasured. It strictly +- `egospeak-omni-zero-residual` has a user-reported seed-0 screen. Causal-valid + dense mean mAP increases from `61.193` to `62.714`; raw onset/offset mean + AP increases from `24.491/38.024` to `25.091/40.604`. Offset improves at all + ten horizons; onset gains are modest. Actual checkpoint provenance and + counts were not supplied for independent verification. Opportunity-based + semantic evaluation is temporarily paused because the user suspects + provisional annotation issues; historical metrics remain archived and P0 + remains unfinished. See the + [`result record`](results/easycom_egospeak_omni_residual_2026-09-16/README.md). + The implementation strictly loads and freezes the released MiniROADA A+V/GRU/anticipation path, reuses existing causal 5 Hz Omni readouts, adds an explicit ten-bin partial-tail phase embedding, and learns only a zero-output residual over the original diff --git a/results/README.md b/results/README.md index e68b606..70a5309 100644 --- a/results/README.md +++ b/results/README.md @@ -6,6 +6,7 @@ readers to the right comparison. | Result record | Question answered | |---|---| +| [`easycom_egospeak_omni_residual_2026-09-16/`](easycom_egospeak_omni_residual_2026-09-16/README.md) | Single-seed frozen EgoSpeak + Omni residual dense/raw endpoint gains; semantic annotation evaluation temporarily paused | | [`easycom_omni_floor_screen_2026-09-16/`](easycom_omni_floor_screen_2026-09-16/README.md) | Whether G+A, pooled-V, or current-frame F1 improve raw boundaries and provisional semantic opportunities over EgoSpeak | | [`easycom_egospeak_three_level_eval_2026-09-16/`](easycom_egospeak_three_level_eval_2026-09-16/README.md) | How dense forecasting, raw boundary anticipation, and provisional opportunity-conditioned ENTRY/YIELD differ for the EgoSpeak baseline | | [easycom_oracle_pooled_wearer_audit_2026-09-10/](easycom_oracle_pooled_wearer_audit_2026-09-10/README.md) | Fixed-four-epoch three-seed oracle test results and the limited wearer gains of historical pooled AV | diff --git a/results/easycom_egospeak_omni_residual_2026-09-16/README.md b/results/easycom_egospeak_omni_residual_2026-09-16/README.md new file mode 100644 index 0000000..3082b6e --- /dev/null +++ b/results/easycom_egospeak_omni_residual_2026-09-16/README.md @@ -0,0 +1,71 @@ +# Stage-one frozen EgoSpeak + Omni residual screen + +## Provenance and scope + +These values were reported by the user on 2026-09-16, not recomputed on this +host. The preceding launcher command specifies seed 0, four fixed epochs on +EasyCom Sessions 4--12, and final evaluation on Sessions 1--3. The reported +run is `results/easycom_egospeak_omni_residual_seed0/checkpoint_last.pt`; its +actual checkpoint hash/config and sample counts were not supplied and remain +unverified. The base is the released `miniroad_easycom_AV.pth` checkpoint. +Implementation: [`stage-one runbook`](../../docs/easycom_egospeak_omni_residual.md). + +The original MiniROADA A+V path and Omni backbone remain frozen; only the +tail-phase-aware residual fusion is trained. Existing Omni states use the +8-s open-stream configuration and a fixed mean over the last-four stack. +Both full-timeline and causal-valid dense numbers are retained in +[`metrics.csv`](metrics.csv). All numbers in this record are percentages; +deltas are percentage points. Displayed values are rounded console output. + +## Findings + +| Metric | Frozen EgoSpeak | Fused | Delta | +|---|---:|---:|---:| +| Causal-valid 0.2-s dense mAP | 74.204 | 77.739 | +3.535 | +| Causal-valid 0.2-s wearer AP | 63.750 | 72.716 | +8.966 | +| Causal-valid ten-horizon dense mean mAP | 61.193 | 62.714 | +1.521 | +| Full-timeline ten-horizon dense mean mAP | 61.155 | 62.676 | +1.521 | +| Ten-horizon raw onset endpoint AP | 24.491 | 25.091 | +0.600 | +| Ten-horizon raw offset endpoint AP | 38.024 | 40.604 | +2.580 | + +Offset AP improves at all ten horizons, by about 2.8--3.5 points at +0.4--1.2 s. Onset AP improves at nine horizons; its largest gain is +1.634 +points at 0.8 s, while long-horizon differences are small. This is a +single-seed positive screen, strongest for future wearer cessation. It is +not a significance test or evidence of semantic floor-yield understanding. +A parameter-matched EgoSpeak-only residual control and additional seeds +remain necessary to attribute the gain specifically to Omni. + +## Raw anticipation definition + +For each horizon h=0.2,...,2.0 s, use valid source/target pairs from the same +session. Onset is evaluated only when wearer activity W(t)=0: positive if +W(t+h)=1, negative if W(t+h)=0, with score p_wearer(t,h). Offset is evaluated +only when W(t)=1: positive if W(t+h)=0, negative if W(t+h)=1, with score +1-p_wearer(t,h). AP ranks all eligible samples pooled over Sessions 1--3; +the headline averages ten horizon APs. This is endpoint state-change +anticipation, not one-event-one-vote timestamp detection. Intermediate +transitions can be invisible when source and endpoint states agree. + +The evaluator also computed exploratory/in-sample operating points, but +only threshold-free endpoint APs are recorded here. They need no Session-4 +threshold calibration. Do not use the accompanying in-sample rate columns +as confirmatory test results. + +## Annotation decision and next stage + +The user suspects issues with the provisional LLM-assisted semantic +annotations. Opportunity-conditioned ENTRY/YIELD evaluation is therefore +temporarily paused pending an annotation audit. Historical provisional +metrics remain archived; they are neither human-reviewed ground truth nor +discarded because of model performance. No new semantic result or semantic +loss is introduced in this screen. + +The next requested implementation is selective MiniROADA GRU/anticipation +fine-tuning from the stage-one checkpoint, keeping original feature +extractors and Omni frozen. This is an exploratory continuation of the +previously documented route, not passage of the still-unfinished P0 semantic +evaluation gate. Do not tune windows, loss weights or stopping on Sessions +1--3; compare fixed budgets and preserve the frozen stage-one/original base +as references. A four-epoch continuation has eight total adaptation epochs +and is not a matched-budget comparison against the four-epoch first stage. diff --git a/results/easycom_egospeak_omni_residual_2026-09-16/metrics.csv b/results/easycom_egospeak_omni_residual_2026-09-16/metrics.csv new file mode 100644 index 0000000..61f0199 --- /dev/null +++ b/results/easycom_egospeak_omni_residual_2026-09-16/metrics.csv @@ -0,0 +1,21 @@ +run,horizon_seconds,dense_full_map,dense_causal_map,dense_causal_background_ap,dense_causal_wearer_ap,dense_causal_others_ap,onset_endpoint_ap,offset_endpoint_ap +egospeak,0.2,74.124,74.204,68.189,63.750,90.673,5.139,9.696 +egospeak,0.4,68.291,68.349,58.596,59.595,86.857,9.878,18.846 +egospeak,0.6,64.267,64.313,52.373,56.619,83.947,15.389,27.730 +egospeak,0.8,61.091,61.128,47.629,54.172,81.585,21.093,34.971 +egospeak,1.0,59.734,59.768,46.172,52.916,80.216,27.225,40.646 +egospeak,1.2,58.774,58.805,45.840,51.488,79.088,30.997,45.112 +egospeak,1.4,57.471,57.498,43.587,50.765,78.143,33.563,47.880 +egospeak,1.6,57.064,57.090,43.518,49.739,78.012,34.046,50.444 +egospeak,1.8,55.933,55.954,42.227,48.605,77.031,34.030,52.051 +egospeak,2.0,54.802,54.821,40.349,47.393,76.720,33.550,52.862 +fused,0.2,77.658,77.739,69.105,72.716,91.396,5.722,11.756 +fused,0.4,71.510,71.570,59.729,67.888,87.093,10.749,21.920 +fused,0.6,66.855,66.903,53.297,63.553,83.858,16.613,31.248 +fused,0.8,63.212,63.251,48.874,59.713,81.167,22.727,38.383 +fused,1.0,60.961,60.996,46.574,56.953,79.460,27.946,43.741 +fused,1.2,59.410,59.441,45.243,54.917,78.163,31.260,47.922 +fused,1.4,58.375,58.404,43.983,53.446,77.782,33.502,50.269 +fused,1.6,57.336,57.361,42.934,51.712,77.439,34.053,52.173 +fused,1.8,56.222,56.244,41.284,50.607,76.840,34.202,53.814 +fused,2.0,55.216,55.235,39.820,49.470,76.415,34.137,54.812 diff --git a/results/easycom_egospeak_omni_residual_2026-09-16/summary.json b/results/easycom_egospeak_omni_residual_2026-09-16/summary.json new file mode 100644 index 0000000..98a9b53 --- /dev/null +++ b/results/easycom_egospeak_omni_residual_2026-09-16/summary.json @@ -0,0 +1,23 @@ +{ + "date": "2026-09-16", + "source": "user-reported console output; not locally recomputed", + "units": "percent; deltas are percentage points", + "split": {"train": [4, 5, 6, 7, 8, 9, 10, 11, 12], "test": [1, 2, 3]}, + "seed": 0, + "epochs": 4, + "configuration_provenance": "preceding command; actual checkpoint/config not inspected", + "checkpoint": "results/easycom_egospeak_omni_residual_seed0/checkpoint_last.pt", + "checkpoint_sha256": null, + "base_checkpoint": "third_party/EgoSpeak/MiniROAD/checkpoints/miniroad_easycom_AV.pth", + "omni_memory_seconds": 8, + "omni_layer_fusion": "last-four fixed mean", + "num_seeds": 1, + "sample_counts": null, + "metrics": { + "egospeak": {"dense_causal_02_map": 74.204, "dense_causal_02_wearer_ap": 63.750, "dense_causal_mean_map": 61.193, "dense_full_mean_map": 61.155, "onset_mean_ap": 24.491, "offset_mean_ap": 38.024}, + "fused": {"dense_causal_02_map": 77.739, "dense_causal_02_wearer_ap": 72.716, "dense_causal_mean_map": 62.714, "dense_full_mean_map": 62.676, "onset_mean_ap": 25.091, "offset_mean_ap": 40.604}, + "fused_minus_egospeak": {"dense_causal_02_map": 3.535, "dense_causal_02_wearer_ap": 8.966, "dense_causal_mean_map": 1.521, "dense_full_mean_map": 1.521, "onset_mean_ap": 0.600, "offset_mean_ap": 2.580} + }, + "semantic_evaluation": "paused: provisional annotation quality requires audit", + "claim_boundary": "single-seed dense/endpoint improvement; not semantic ENTRY/YIELD or Omni-specific causal attribution" +}