Fix opportunity competing-risk evaluation

This commit is contained in:
Xin Wang
2026-09-16 04:31:18 +08:00
parent 02e83c0c11
commit d4c900ab13
10 changed files with 310 additions and 88 deletions
+16 -13
View File
@@ -54,19 +54,23 @@ result.
- An opportunity-conditioned dense-output evaluator aligns ENTRY positives and - An opportunity-conditioned dense-output evaluator aligns ENTRY positives and
core negatives at non-wearer releases, aligns INT positives and hold-floor core negatives at non-wearer releases, aligns INT positives and hold-floor
negatives at competing non-wearer onsets, and applies strict competing-risk negatives at competing non-wearer onsets, and applies strict competing-risk
censoring. It uses only the original wearer probabilities, excludes wearer censoring. ENTRY positives whose selected release is followed by a competing
backchannels from headline ENTRY FPR, and records unanchored/ambiguous cases. other onset or intervening wearer activity are excluded, and ambiguous
multi-competitor INT cases are not assigned an arbitrary onset. It uses only
the original wearer probabilities, excludes wearer backchannels from
headline ENTRY FPR, and records unanchored/ambiguous cases.
### Measured screening result ### Measured screening result
- The released EgoSpeak MiniROADA A+V checkpoint on Sessions 1--3 obtains - The released EgoSpeak MiniROADA A+V checkpoint on Sessions 1--3 obtains
`74.124` 0.2-second mAP and `61.155` ten-horizon mean mAP. Raw wearer `74.124` 0.2-second mAP and `61.155` ten-horizon mean mAP. Raw wearer
onset/offset anticipation is harder (`24.491/38.024` mean AP). On provisional onset/offset anticipation is harder (`24.491/38.024` mean AP). On provisional
LLM-assisted matched opportunities, ENTRY AP is `52.487` versus `32.731%` LLM-assisted matched opportunities, the original v1 evaluator reported
prevalence, while INT/YIELD AP is `20.135`, indistinguishable from its ENTRY/INT AP of `52.487/20.135`; those opportunity values are superseded
`20.438%` prevalence. Figure-7-style continuous wearer curves therefore because v1 could cross a competing ENTRY onset and arbitrarily selected among
support dense forecasting, not semantic floor-control claims. Full curves, multiple INT competitors. Figure-7-style continuous wearer curves support
counts, exclusions, and caveats are frozen in dense forecasting, not semantic floor-control claims. The unaffected dense
curves and historical v1 opportunity record are in
[`results/easycom_egospeak_three_level_eval_2026-09-16/`](results/easycom_egospeak_three_level_eval_2026-09-16/README.md). [`results/easycom_egospeak_three_level_eval_2026-09-16/`](results/easycom_egospeak_three_level_eval_2026-09-16/README.md).
- F1 Frame-set Attention uses fused G+A as one query over the latest half-open - F1 Frame-set Attention uses fused G+A as one query over the latest half-open
causal person set. On the existing three-seed, four-epoch test screen it causal person set. On the existing three-seed, four-epoch test screen it
@@ -74,12 +78,11 @@ result.
`64.95099 +/- 0.09451` ten-horizon mean mAP. This is stable, but is not a `64.95099 +/- 0.09451` ten-horizon mean mAP. This is stable, but is not a
matched improvement over the historical no-dev pooled result. Its three-seed matched improvement over the historical no-dev pooled result. Its three-seed
boundary response/yield AP is `28.423 +/- 0.480` / `48.641 +/- 0.391`; on boundary response/yield AP is `28.423 +/- 0.480` / `48.641 +/- 0.391`; on
provisional reviewed opportunities its ENTRY/INT AP is the superseded v1 opportunity evaluator reported ENTRY/INT AP of
`41.561 +/- 0.581` / `22.991 +/- 0.202`. These values are respectively `41.561 +/- 0.581` / `22.991 +/- 0.202`; these values require a v2 rerun and
`-4.461/-1.275` points below the historical pooled-V seed-0 result, so the cannot currently support a semantic-selectivity conclusion. A same-split
current F1 screen does not support the hoped-for semantic-selectivity gain. three-seed pooled control and correct/wrong visual controls remain missing.
A same-split three-seed pooled control and correct/wrong visual controls See the historical
remain missing. See the immutable
[`screening record`](results/easycom_omni_floor_screen_2026-09-16/README.md). [`screening record`](results/easycom_omni_floor_screen_2026-09-16/README.md).
### Measured local-fusion screens ### Measured local-fusion screens
+23 -14
View File
@@ -10,6 +10,12 @@ head.
A positive must have a reviewer-selected preceding non-wearer release. Its A positive must have a reviewer-selected preceding non-wearer release. Its
opportunity starts at that release and resolves at the reviewed wearer entry. opportunity starts at that release and resolves at the reviewed wearer entry.
If a new non-wearer onset occurs before or exactly at that entry, the release
has already been resolved by a competing event and the positive is excluded.
The same applies when wearer activity already occurred between the selected
release and the reviewed onset: the selected release is then a stale,
post-outcome anchor. Both cases are counted separately in
`review_diagnostics.skipped`.
Core hard negatives start at the same type of non-wearer release and resolve Core hard negatives start at the same type of non-wearer release and resolve
when the same speaker resumes (`ENTRY_OTHER_MIDTURN_PAUSE`) or a different when the same speaker resumes (`ENTRY_OTHER_MIDTURN_PAUSE`) or a different
non-wearer takes the next floor (`ENTRY_OTHER_TO_OTHER`). The score is the non-wearer takes the next floor (`ENTRY_OTHER_TO_OTHER`). The score is the
@@ -27,10 +33,10 @@ the maximum `1 - p_wearer` over 0.2--0.6 seconds, censored strictly at that
non-wearer's offset. Reviewed `INT_YIELD` items are positive; other non-wearer's offset. Reviewed `INT_YIELD` items are positive; other
backchannels, failed takeovers, and clear wearer-hold cases are hard negatives. backchannels, failed takeovers, and clear wearer-hold cases are hard negatives.
Current INT reviews do not select one competing onset. The loader therefore Current INT reviews do not select one competing onset. A unique supplied other
uses the latest supplied other interval that crosses the wearer offset within interval that crosses the wearer offset within the annotation tolerance is
the annotation tolerance. It records how many intervals were eligible so used. Cases with multiple eligible intervals are excluded and counted rather
multi-competitor cases can be audited before the protocol is frozen. than being assigned a deterministic latest-onset anchor.
## Provisional EgoSpeak run ## Provisional EgoSpeak run
@@ -58,21 +64,24 @@ Before manuscript reporting, freeze the 2.0-second ENTRY cap, 0.6-second YIELD
cap, positive anchor-selection policy, and multiple-competitor handling using cap, positive anchor-selection policy, and multiple-competitor handling using
training-side data only. training-side data only.
## Recorded Sessions 1--3 baseline ## Superseded v1 Sessions 1--3 baseline
The first threshold-free run gives ENTRY AP 52.487% (163 positive, 335 hard The first threshold-free run used the superseded v1 loader and reported ENTRY
negative; 32.731% prevalence) and INT/YIELD AP 20.135% (28 positive, 109 hard AP 52.487% (163 positive, 335 hard negative; 32.731% prevalence) and INT/YIELD
negative; 20.438% prevalence). It excludes 81 ENTRY and 5 INT cases as invalid AP 20.135% (28 positive, 109 hard negative; 20.438% prevalence). It excluded
or post-outcome. These are provisional LLM-assisted labels, not human-reviewed 81 ENTRY and 5 INT cases as invalid or post-outcome. These opportunity values
confirmatory results. Dense, raw-boundary, and opportunity metrics and their must not be reported: v1 allowed a
Figure 7 interpretation are frozen together in the selected ENTRY release to cross an intervening other onset and selected the
latest onset in ambiguous multi-competitor INT cases. Dense and raw-boundary
metrics are unaffected. Rerun the opportunity evaluation with protocol v2
before updating the manuscript. The historical output is retained in the
[`2026-09-16 result record`](../results/easycom_egospeak_three_level_eval_2026-09-16/README.md). [`2026-09-16 result record`](../results/easycom_egospeak_three_level_eval_2026-09-16/README.md).
The expanded Omni screen finds no opportunity-ranking gain from the current The expanded v1 Omni screen found no opportunity-ranking gain from the current
F1 Frame-set Attention model: its three-seed ENTRY/INT AP is F1 Frame-set Attention model: its three-seed ENTRY/INT AP is
41.561%/22.991%, below the historical pooled-V seed-0 values 41.561%/22.991%, below the historical pooled-V seed-0 values
46.022%/24.266%. Because F1 and that pooled run use different train/dev model- 46.022%/24.266%. Because F1 and that pooled run use different train/dev model-
selection protocols, this is a stable negative screen rather than a matched selection protocols and used the superseded opportunity loader, only its dense
confirmatory comparison. Dense, boundary, per-seed, and provenance details are and raw-boundary findings remain current. Per-seed and provenance details are
in the in the
[`Omni floor screening record`](../results/easycom_omni_floor_screen_2026-09-16/README.md). [`Omni floor screening record`](../results/easycom_omni_floor_screen_2026-09-16/README.md).
@@ -168,12 +168,13 @@ forecasting 的定性解释,不能代替 boundary 或 semantic opportunity 评
曲线和限制见 曲线和限制见
[`three-level baseline record`](../results/easycom_egospeak_three_level_eval_2026-09-16/README.md)。 [`three-level baseline record`](../results/easycom_egospeak_three_level_eval_2026-09-16/README.md)。
当前 F1 current-frame person-set screen 的三种子结果同样没有支持 semantic floor 当前 F1 current-frame person-set screen 的 dense mean mAP 为
selectivity:ENTRY/INT opportunity AP 为 `41.561 +/- 0.581` / `22.991 +/- 0.202`, `64.951 +/- 0.095`,没有超过历史 pooled-V 的 `65.400`。原 v1 evaluator 给出的
低于历史 pooled-V seed-0 的 `46.022/24.266`;dense mean mAP 也为 ENTRY/INT opportunity AP(`41.561 +/- 0.581` / `22.991 +/- 0.202`)已失效:v1
`64.951 +/- 0.095`,没有超过历史 pooled-V 的 `65.400`。由于两者 train/dev 可能跨过 competing ENTRY onset,且会在多个 INT competitor 中任意选择 latest onset。
选择协议不同,这只能判为稳定的负向 screening,正式归因前必须补同协议三种子 pooled 必须用 v2 重跑后才能判断 semantic floor selectivity;正式归因还必须补同协议三种子
control。详见 [`Omni floor screening record`](../results/easycom_omni_floor_screen_2026-09-16/README.md)。 pooled control。历史数值与勘误见
[`Omni floor screening record`](../results/easycom_omni_floor_screen_2026-09-16/README.md)。
### P3:第一优先方法——匿名逐人时序旁路 ### P3:第一优先方法——匿名逐人时序旁路
+136 -25
View File
@@ -8,14 +8,18 @@ event head.
ENTRY opportunities are anchored at a reviewed preceding non-wearer release. ENTRY opportunities are anchored at a reviewed preceding non-wearer release.
Positive opportunities resolve at the wearer entry; core hard negatives Positive opportunities resolve at the wearer entry; core hard negatives
resolve when the same or a different non-wearer speaks next. The score is the resolve when the same or a different non-wearer speaks next. A positive is
maximum wearer probability over available 0.2--2.0 s forecasts up to that excluded if another non-wearer onset competes before (or exactly at) its wearer
competing resolution. entry, or if wearer activity already occurred after the selected release. The
score is the maximum wearer probability over available 0.2--2.0 s forecasts up
to the uncontested resolution.
YIELD opportunities are anchored at a non-wearer onset during wearer speech. YIELD opportunities are anchored at a non-wearer onset during wearer speech.
The score is the maximum future wearer-inactivity probability over 0.2--0.6 s, The score is the maximum future wearer-inactivity probability over 0.2--0.6 s,
censored at that non-wearer's offset. Positive and hard-negative examples censored at that non-wearer's offset. Reviews with multiple eligible
therefore begin at the same kind of observed conversational opportunity. competing onsets are excluded until the annotation selects one. Positive and
hard-negative examples therefore begin at the same kind of unambiguous
observed conversational opportunity.
Prediction arrays use the repository contract Prediction arrays use the repository contract
``[target_time, class, future_horizon]``. All scores are read diagonally from ``[target_time, class, future_horizon]``. All scores are read diagonally from
@@ -178,17 +182,83 @@ def _candidate_region(
return start, end return start, end
def _int_competing_interval( def _entry_selected_option(
candidate: dict[str, Any],
review: dict[str, Any],
*,
selected_release_seconds: float,
uid: str,
) -> dict[str, Any] | None:
evidence = candidate.get("temporal_evidence")
evidence = evidence if isinstance(evidence, dict) else {}
options = evidence.get("opportunity_options")
if not isinstance(options, list):
return None
selected_uid = review.get("selected_opportunity_uid")
matches: list[dict[str, Any]] = []
for index, option in enumerate(options):
if not isinstance(option, dict):
continue
option_seconds = _optional_seconds(
option.get("anchor_seconds"),
field=f"opportunity_options[{index}].anchor_seconds",
uid=uid,
)
if selected_uid is not None and option.get("uid") == selected_uid:
matches.append(option)
elif selected_uid is None and option_seconds is not None and math.isclose(
option_seconds, selected_release_seconds, abs_tol=1e-7
):
matches.append(option)
if len(matches) > 1:
raise ValueError(f"{uid}: selected ENTRY opportunity is not unique")
return matches[0] if matches else None
def _entry_competing_onsets(
candidate: dict[str, Any],
*,
selected_release_seconds: float,
behavior_anchor_seconds: float,
uid: str,
) -> list[float]:
"""Return other onsets that resolve a release before the wearer entry."""
evidence = candidate.get("temporal_evidence")
evidence = evidence if isinstance(evidence, dict) else {}
raw_intervals = evidence.get("gap_other_intervals")
if not isinstance(raw_intervals, list):
raw_intervals = []
onsets: set[float] = set()
for index, interval in enumerate(raw_intervals):
if not isinstance(interval, dict):
continue
onset = _optional_seconds(
interval.get("onset_seconds"),
field=f"gap_other_intervals[{index}].onset_seconds",
uid=uid,
)
if onset is None:
continue
if (
onset >= selected_release_seconds - 1e-7
and onset <= behavior_anchor_seconds + 1e-7
):
onsets.add(onset)
return sorted(onsets)
def _int_competing_intervals(
candidate: dict[str, Any], candidate: dict[str, Any],
*, *,
behavior_anchor_seconds: float, behavior_anchor_seconds: float,
uid: str, uid: str,
) -> tuple[float, float, int] | None: ) -> list[tuple[float, float]]:
evidence = candidate.get("temporal_evidence") evidence = candidate.get("temporal_evidence")
evidence = evidence if isinstance(evidence, dict) else {} evidence = evidence if isinstance(evidence, dict) else {}
raw_intervals = evidence.get("nearby_other_intervals") raw_intervals = evidence.get("nearby_other_intervals")
if not isinstance(raw_intervals, list): if not isinstance(raw_intervals, list):
return None return []
eligible: list[tuple[float, float]] = [] eligible: list[tuple[float, float]] = []
for index, interval in enumerate(raw_intervals): for index, interval in enumerate(raw_intervals):
if not isinstance(interval, dict): if not isinstance(interval, dict):
@@ -209,13 +279,7 @@ def _int_competing_interval(
continue continue
if onset < behavior_anchor_seconds and offset > onset: if onset < behavior_anchor_seconds and offset > onset:
eligible.append((onset, offset)) eligible.append((onset, offset))
if not eligible: return sorted(set(eligible))
return None
# A semantic INT review does not currently select one competing onset.
# The latest interval crossing the wearer offset is the deterministic,
# most proximal causal candidate. Multiplicity is surfaced for audit.
onset, offset = max(eligible, key=lambda item: (item[0], item[1]))
return onset, offset, len(eligible)
def load_opportunity_events( def load_opportunity_events(
@@ -278,6 +342,38 @@ def load_opportunity_events(
if selected_release is None: if selected_release is None:
skipped["entry_positive_without_selected_release"] += 1 skipped["entry_positive_without_selected_release"] += 1
continue continue
selected_option = _entry_selected_option(
candidate,
review,
selected_release_seconds=selected_release,
uid=uid,
)
if (
selected_option is not None
and selected_option.get(
"wearer_active_between_release_and_onset"
) is True
):
skipped["entry_positive_post_outcome_wearer_activity"] += 1
continue
competing_onsets = _entry_competing_onsets(
candidate,
selected_release_seconds=selected_release,
behavior_anchor_seconds=behavior,
uid=uid,
)
option_reports_competitor = (
selected_option is not None
and selected_option.get(
"other_onset_between_release_and_onset"
) is True
)
if option_reports_competitor and not competing_onsets:
skipped["entry_positive_competing_onset_unresolved"] += 1
continue
if competing_onsets:
skipped["entry_positive_competed_before_behavior"] += 1
continue
anchor = selected_release anchor = selected_release
resolution = behavior resolution = behavior
censor = behavior censor = behavior
@@ -298,17 +394,21 @@ def load_opportunity_events(
competing_intervals = 0 competing_intervals = 0
else: else:
if positive: if positive:
interval = _int_competing_interval( intervals = _int_competing_intervals(
candidate, candidate,
behavior_anchor_seconds=behavior, behavior_anchor_seconds=behavior,
uid=uid, uid=uid,
) )
if interval is None: if not intervals:
skipped["int_positive_without_crossing_other"] += 1 skipped["int_positive_without_crossing_other"] += 1
continue continue
anchor, censor, competing_intervals = interval if len(intervals) > 1:
skipped["int_positive_ambiguous_competing_others"] += 1
continue
anchor, censor = intervals[0]
competing_intervals = 1
resolution = behavior resolution = behavior
basis = "latest_other_onset_crossing_wearer_offset" basis = "unique_other_onset_crossing_wearer_offset"
elif subtype_value in INT_HARD_NEGATIVES: elif subtype_value in INT_HARD_NEGATIVES:
if source in INT_MINED_SOURCES: if source in INT_MINED_SOURCES:
anchor, censor = _candidate_region(candidate, uid=uid) anchor, censor = _candidate_region(candidate, uid=uid)
@@ -316,17 +416,21 @@ def load_opportunity_events(
competing_intervals = 1 competing_intervals = 1
basis = "mined_competing_other_onset" basis = "mined_competing_other_onset"
else: else:
interval = _int_competing_interval( intervals = _int_competing_intervals(
candidate, candidate,
behavior_anchor_seconds=behavior, behavior_anchor_seconds=behavior,
uid=uid, uid=uid,
) )
if interval is None: if not intervals:
skipped["int_hn_without_competing_other"] += 1 skipped["int_hn_without_competing_other"] += 1
continue continue
anchor, censor, competing_intervals = interval if len(intervals) > 1:
skipped["int_hn_ambiguous_competing_others"] += 1
continue
anchor, censor = intervals[0]
competing_intervals = 1
resolution = behavior resolution = behavior
basis = "latest_other_onset_crossing_non_yield_offset" basis = "unique_other_onset_crossing_non_yield_offset"
else: else:
skipped["int_unsupported_hard_negative"] += 1 skipped["int_unsupported_hard_negative"] += 1
continue continue
@@ -731,8 +835,8 @@ def main() -> None:
else calibration_diagnostics["num_provisional_llm"] else calibration_diagnostics["num_provisional_llm"]
) )
payload = { payload = {
"schema_version": 1, "schema_version": 2,
"protocol": "egospeak-opportunity-conditioned-floor-actions-v1", "protocol": "egospeak-opportunity-conditioned-floor-actions-v2",
"semantic_scope": ( "semantic_scope": (
"LLM-assisted opportunity labels with dense wearer-activity scores; " "LLM-assisted opportunity labels with dense wearer-activity scores; "
"this is provisional and no human review is implied." "this is provisional and no human review is implied."
@@ -757,6 +861,13 @@ def main() -> None:
"yield_max_seconds": yield_max, "yield_max_seconds": yield_max,
"entry_headline_hard_negatives": sorted(ENTRY_CORE_HARD_NEGATIVES), "entry_headline_hard_negatives": sorted(ENTRY_CORE_HARD_NEGATIVES),
"entry_backchannel_policy": "separate diagnostic; excluded from headline", "entry_backchannel_policy": "separate diagnostic; excluded from headline",
"entry_competing_risk_policy": (
"exclude selected-release positives with an other onset at or before "
"the wearer entry, or with intervening wearer activity"
),
"int_multiple_competitor_policy": (
"exclude until one competing onset is selected by review"
),
"matched_recalls": recalls, "matched_recalls": recalls,
"review_diagnostics": review_diagnostics, "review_diagnostics": review_diagnostics,
"calibration_review_diagnostics": calibration_diagnostics, "calibration_review_diagnostics": calibration_diagnostics,
+6 -7
View File
@@ -406,8 +406,8 @@ verify identical targets, causal validity, and evaluation.}
\begin{table}[t] \begin{table}[t]
\caption{Confirmatory EasyCom results. Entry and INT false-positive rates use \caption{Confirmatory EasyCom results. Entry and INT false-positive rates use
the opportunity-conditioned activity score at matched recall $R=0.8$; E/I AP the opportunity-conditioned activity score at matched recall $R=0.8$; E/I AP
is the corresponding opportunity AP. $\dagger$ denotes provisional is the corresponding opportunity AP. Event results will be populated only
LLM-assisted labels pending human review.} after protocol-v2 reruns and human adjudication.}
\label{tab:main} \label{tab:main}
\centering \centering
\small \small
@@ -417,7 +417,7 @@ LLM-assisted labels pending human review.}
Method & 0.2-s mAP $\uparrow$ & Mean mAP $\uparrow$ Method & 0.2-s mAP $\uparrow$ & Mean mAP $\uparrow$
& Entry FPR@$R=.8$ $\downarrow$ & INT FPR@$R=.8$ $\downarrow$ & E/I AP $\uparrow$\\ & Entry FPR@$R=.8$ $\downarrow$ & INT FPR@$R=.8$ $\downarrow$ & E/I AP $\uparrow$\\
\hline \hline
EgoSpeak MiniROADA A+V & 74.12 & 61.16 & \todo{--} & \todo{--} & $52.49/20.13^\dagger$\\ EgoSpeak MiniROADA A+V & 74.12 & 61.16 & \todo{--} & \todo{--} & \todo{--/--}\\
Omni-3B frozen $G$ & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\ Omni-3B frozen $G$ & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
Omni-3B matched PEFT & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\ Omni-3B matched PEFT & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
$G+$local audio & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\ $G+$local audio & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
@@ -430,10 +430,9 @@ Omni-7B scaling control & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \t
\textbf{Dense forecasting.} \textbf{Dense forecasting.}
The EgoSpeak baseline reaches 74.12\% 0.2-s mAP and 61.16\% mean mAP; its raw The EgoSpeak baseline reaches 74.12\% 0.2-s mAP and 61.16\% mean mAP; its raw
wearer-onset/offset mean APs are 24.49/38.02\%. On provisional reviewed wearer-onset/offset mean APs are 24.49/38.02\%. Opportunity-conditioned values
opportunities, ENTRY AP is 52.49\% versus 32.73\% prevalence, whereas INT AP is remain pending a protocol-v2 rerun and human review; continuous wearer-score
20.13\% versus 20.44\%; continuous wearer-score plots therefore do not alone plots do not alone establish semantic floor-yield selectivity.
establish semantic floor-yield selectivity.
The proposed local AV adapter improves 0.2-s mAP from The proposed local AV adapter improves 0.2-s mAP from
\todo{$X$ to $Y$} over the strongest matched baseline and improves \todo{$X$ to $Y$} over the strongest matched baseline and improves
mean mAP by \todo{$Z$ points}. mean mAP by \todo{$Z$ points}.
@@ -1,5 +1,11 @@
# EasyCom EgoSpeak three-level evaluation # EasyCom EgoSpeak three-level evaluation
> **Protocol erratum.** The dense and raw-boundary results in this record remain
> valid. The opportunity-conditioned ENTRY/INT values were produced by the
> superseded v1 loader, which could cross an intervening ENTRY competitor and
> selected the latest onset in ambiguous multi-competitor INT cases. Do not cite
> those opportunity values; rerun protocol v2 and create a new result record.
This record freezes the first Sessions 1--3 evaluation of the released This record freezes the first Sessions 1--3 evaluation of the released
EgoSpeak MiniROADA audio-visual checkpoint under three deliberately separate EgoSpeak MiniROADA audio-visual checkpoint under three deliberately separate
questions. The values below were produced on the remote experiment host; the questions. The values below were produced on the remote experiment host; the
@@ -15,7 +21,7 @@ prediction and per-event artifacts remain on that host and are not committed.
- Raw boundary anticipation: threshold-free AP for wearer `OFF->ON` and - Raw boundary anticipation: threshold-free AP for wearer `OFF->ON` and
`ON->OFF` changes. These are timing-derived diagnostics, not semantic EOT or `ON->OFF` changes. These are timing-derived diagnostics, not semantic EOT or
INT claims. INT claims.
- Opportunity-conditioned behavior: one causally censored activity-derived - Opportunity-conditioned behavior: historical v1 activity-derived
score per reviewed ENTRY or INT/YIELD opportunity. Labels are provisional score per reviewed ENTRY or INT/YIELD opportunity. Labels are provisional
two-pass LLM-assisted reviews and have not been human reviewed. two-pass LLM-assisted reviews and have not been human reviewed.
- No test-set threshold is fitted for any AP reported here. Session 4 is needed - No test-set threshold is fitted for any AP reported here. Session 4 is needed
@@ -34,15 +40,16 @@ mean AP, respectively. These lower values do not contradict the dense result:
dense AP rewards correct future state over sustained regions, whereas a raw dense AP rewards correct future state over sustained regions, whereas a raw
boundary event is counted once and has many harder non-boundary comparisons. boundary event is counted once and has many harder non-boundary comparisons.
On the provisional opportunity labels, ENTRY AP is **52.487%** for 163 The superseded v1 run reported ENTRY AP of **52.487%** for 163
positives and 335 hard negatives, versus 32.731% positive prevalence positives and 335 hard negatives, versus 32.731% positive prevalence
(+19.756 points; 1.60x prevalence). INT/YIELD AP is **20.135%** for 28 positives (+19.756 points; 1.60x prevalence). INT/YIELD AP is **20.135%** for 28 positives
and 109 hard negatives, essentially equal to its 20.438% prevalence. This is and 109 hard negatives, essentially equal to its 20.438% prevalence. These
consistent with the output contract: wearer probability can rank many entry values are retained only for provenance and must be replaced by a v2 rerun. The
opportunities, but `1-p(wearer)` does not encode whether an interruption should original pattern was consistent with the output contract: wearer probability
cause a yield. The ENTRY and INT loaders additionally exclude 81 and 5 can rank many entry opportunities, but `1-p(wearer)` does not encode whether an
invalid/post-outcome cases; these exclusions require an audit before any final interruption should cause a yield. The ENTRY and INT loaders additionally
paper claim. exclude 81 and 5 invalid/post-outcome cases; these exclusions require an audit
before any final paper claim.
## Relation to EgoSpeak Figure 7 ## Relation to EgoSpeak Figure 7
@@ -73,4 +80,3 @@ Machine-readable curves and counts are in [`summary.json`](summary.json) and
the compact comparison is in [`metrics.csv`](metrics.csv). The opportunity the compact comparison is in [`metrics.csv`](metrics.csv). The opportunity
protocol and reproduction command are documented in protocol and reproduction command are documented in
[`docs/easycom_opportunity_event_evaluation.md`](../../docs/easycom_opportunity_event_evaluation.md). [`docs/easycom_opportunity_event_evaluation.md`](../../docs/easycom_opportunity_event_evaluation.md).
@@ -1,6 +1,8 @@
{ {
"recorded_at": "2026-09-16", "recorded_at": "2026-09-16",
"source": "user-provided remote experiment output", "source": "user-provided remote experiment output",
"opportunity_protocol_status": "superseded_v1_rerun_required",
"opportunity_protocol_erratum": "Dense and raw-boundary metrics remain valid; opportunity metrics must be rerun with egospeak-opportunity-conditioned-floor-actions-v2.",
"dataset": "EasyCom", "dataset": "EasyCom",
"sessions": ["Session_1", "Session_2", "Session_3"], "sessions": ["Session_1", "Session_2", "Session_3"],
"model": "EgoSpeak MiniROADA audio-visual released checkpoint", "model": "EgoSpeak MiniROADA audio-visual released checkpoint",
@@ -57,4 +59,3 @@
"threshold_free": true, "threshold_free": true,
"session4_required_for_reported_metrics": false "session4_required_for_reported_metrics": false
} }
@@ -1,5 +1,12 @@
# EasyCom Omni floor-event screening # EasyCom Omni floor-event screening
> **Protocol erratum.** Dense and raw-boundary metrics remain valid. All
> opportunity-conditioned ENTRY/INT values below used the superseded v1 loader,
> which could cross an intervening ENTRY competitor and selected the latest
> onset in ambiguous multi-competitor INT cases. Retain them for provenance
> only; do not use them for model selection or manuscript claims before a v2
> rerun.
This record freezes the first comparison of EgoSpeak, Omni `G+A`, Omni This record freezes the first comparison of EgoSpeak, Omni `G+A`, Omni
`G+A+pooled-V`, and the three-seed current-frame F1 Frame-set Attention screen `G+A+pooled-V`, and the three-seed current-frame F1 Frame-set Attention screen
on EasyCom Sessions 1--3. Values were produced on the remote experiment host on EasyCom Sessions 1--3. Values were produced on the remote experiment host
@@ -11,9 +18,8 @@ CSVs remain on that host and are not committed here.
1. Dense future activity uses causal-valid framewise AP/mAP over ten horizons. 1. Dense future activity uses causal-valid framewise AP/mAP over ten horizons.
2. Raw wearer endpoint and aggregate-other-boundary AP are threshold-free 2. Raw wearer endpoint and aggregate-other-boundary AP are threshold-free
timing diagnostics; they are not semantic EOT/INT labels. timing diagnostics; they are not semantic EOT/INT labels.
3. Opportunity-conditioned ENTRY and INT/YIELD AP assigns one causally 3. Opportunity-conditioned ENTRY and INT/YIELD AP is a historical v1 screen
censored activity-derived score to each provisional LLM-assisted reviewed that requires rerunning with the corrected v2 competing-risk protocol.
opportunity.
All opportunity runs use identical denominators: 163 ENTRY positives and 335 All opportunity runs use identical denominators: 163 ENTRY positives and 335
hard negatives, with 81 invalid/post-outcome cases; and 28 INT positives and hard negatives, with 81 invalid/post-outcome cases; and 28 INT positives and
@@ -39,7 +45,7 @@ from EgoSpeak to G+A to G+A+pooled-V as follows: onset AP
offset/yield-timing gain comes from G+A; pooled visual input adds almost offset/yield-timing gain comes from G+A; pooled visual input adds almost
nothing to raw transition timing. nothing to raw transition timing.
## Interpretation ## Historical v1 interpretation
- G+A strongly improves dense forecasting and generic wearer-offset/yield - G+A strongly improves dense forecasting and generic wearer-offset/yield
timing over EgoSpeak, but not reviewed semantic opportunities. In particular, timing over EgoSpeak, but not reviewed semantic opportunities. In particular,
@@ -77,4 +83,3 @@ nothing to raw transition timing.
Exact aggregate values are stored in [`summary.json`](summary.json) and Exact aggregate values are stored in [`summary.json`](summary.json) and
[`metrics.csv`](metrics.csv). The earlier EgoSpeak-only record is [`metrics.csv`](metrics.csv). The earlier EgoSpeak-only record is
[`easycom_egospeak_three_level_eval_2026-09-16`](../easycom_egospeak_three_level_eval_2026-09-16/README.md). [`easycom_egospeak_three_level_eval_2026-09-16`](../easycom_egospeak_three_level_eval_2026-09-16/README.md).
@@ -1,6 +1,8 @@
{ {
"recorded_at": "2026-09-16", "recorded_at": "2026-09-16",
"source": "user-provided remote experiment output", "source": "user-provided remote experiment output",
"opportunity_protocol_status": "superseded_v1_rerun_required",
"opportunity_protocol_erratum": "Dense and raw-boundary metrics remain valid; opportunity metrics must be rerun with egospeak-opportunity-conditioned-floor-actions-v2.",
"dataset": "EasyCom", "dataset": "EasyCom",
"test_sessions": ["Session_1", "Session_2", "Session_3"], "test_sessions": ["Session_1", "Session_2", "Session_3"],
"scale": "percent", "scale": "percent",
@@ -82,4 +84,3 @@
"threshold_free_metrics_only": true, "threshold_free_metrics_only": true,
"matched_pooled_control_available": false "matched_pooled_control_available": false
} }
+93 -7
View File
@@ -30,7 +30,10 @@ def reviewed_row(
region_start: float, region_start: float,
region_end: float, region_end: float,
selected_release: float | None = None, selected_release: float | None = None,
selected_release_uid: str | None = None,
nearby: list[dict[str, object]] | None = None, nearby: list[dict[str, object]] | None = None,
opportunity_options: list[dict[str, object]] | None = None,
gap_other_intervals: list[dict[str, object]] | None = None,
candidate_source: str | None = None, candidate_source: str | None = None,
) -> dict[str, object]: ) -> dict[str, object]:
event_type = None event_type = None
@@ -45,6 +48,7 @@ def reviewed_row(
"human_verdict": verdict, "human_verdict": verdict,
"human_event_type": event_type, "human_event_type": event_type,
"human_subtype": subtype, "human_subtype": subtype,
"selected_opportunity_uid": selected_release_uid,
"selected_opportunity_seconds": selected_release, "selected_opportunity_seconds": selected_release,
"boundary_correction_seconds": 0, "boundary_correction_seconds": 0,
}, },
@@ -59,6 +63,8 @@ def reviewed_row(
"behavior_anchor_seconds": anchor, "behavior_anchor_seconds": anchor,
"wearer_offset_seconds": anchor, "wearer_offset_seconds": anchor,
"nearby_other_intervals": nearby or [], "nearby_other_intervals": nearby or [],
"opportunity_options": opportunity_options or [],
"gap_other_intervals": gap_other_intervals or [],
}, },
}, },
} }
@@ -119,7 +125,7 @@ class OpportunityEventTests(unittest.TestCase):
diagnostics["skipped"]["entry_backchannel_separate_diagnostic"], 1 diagnostics["skipped"]["entry_backchannel_separate_diagnostic"], 1
) )
def test_loader_selects_latest_crossing_other_for_int(self) -> None: def test_loader_excludes_ambiguous_crossing_others_for_int(self) -> None:
nearby = [ nearby = [
{ {
"onset_seconds": 1.0, "onset_seconds": 1.0,
@@ -145,11 +151,91 @@ class OpportunityEventTests(unittest.TestCase):
) )
]) ])
self.addCleanup(temporary.cleanup) self.addCleanup(temporary.cleanup)
events, _ = load_opportunity_events([path], ["Session_4"]) events, diagnostics = load_opportunity_events([path], ["Session_4"])
self.assertEqual(len(events), 1) self.assertEqual(events, [])
self.assertEqual(events[0].opportunity_anchor_seconds, 2.0) self.assertEqual(
self.assertEqual(events[0].censor_seconds, 3.8) diagnostics["skipped"]["int_positive_ambiguous_competing_others"],
self.assertEqual(events[0].competing_intervals, 2) 1,
)
def test_loader_excludes_entry_positive_at_competing_other_onset(self) -> None:
rows = [
reviewed_row(
"entry-competed",
family="entry",
verdict="POSITIVE",
subtype=None,
anchor=1.6,
region_start=0.0,
region_end=3.0,
selected_release=1.0,
selected_release_uid="release",
opportunity_options=[
{
"uid": "release",
"anchor_seconds": 1.0,
"wearer_active_between_release_and_onset": False,
"other_onset_between_release_and_onset": True,
}
],
gap_other_intervals=[
{"uid": "competitor", "onset_seconds": 1.4}
],
),
reviewed_row(
"entry-simultaneous",
family="entry",
verdict="POSITIVE",
subtype=None,
anchor=4.0,
region_start=3.0,
region_end=5.0,
selected_release=3.2,
gap_other_intervals=[
{"uid": "competitor", "onset_seconds": 4.0}
],
),
]
temporary, path = self.write_reviews(rows)
self.addCleanup(temporary.cleanup)
events, diagnostics = load_opportunity_events([path], ["Session_4"])
self.assertEqual(events, [])
self.assertEqual(
diagnostics["skipped"]["entry_positive_competed_before_behavior"],
2,
)
def test_loader_excludes_entry_positive_after_prior_wearer_activity(self) -> None:
temporary, path = self.write_reviews([
reviewed_row(
"entry-post-outcome",
family="entry",
verdict="POSITIVE",
subtype=None,
anchor=2.0,
region_start=0.0,
region_end=3.0,
selected_release=1.0,
selected_release_uid="release",
opportunity_options=[
{
"uid": "release",
"anchor_seconds": 1.0,
"wearer_active_between_release_and_onset": True,
"other_onset_between_release_and_onset": False,
}
],
)
])
self.addCleanup(temporary.cleanup)
events, diagnostics = load_opportunity_events([path], ["Session_4"])
self.assertEqual(events, [])
self.assertEqual(
diagnostics["skipped"][
"entry_positive_post_outcome_wearer_activity"
],
1,
)
def test_entry_hn_requires_matching_independently_mined_source(self) -> None: def test_entry_hn_requires_matching_independently_mined_source(self) -> None:
rows = [ rows = [
@@ -385,7 +471,7 @@ class OpportunityEventTests(unittest.TestCase):
payload = json.loads(output.read_text(encoding="utf-8")) payload = json.loads(output.read_text(encoding="utf-8"))
self.assertEqual( self.assertEqual(
payload["protocol"], payload["protocol"],
"egospeak-opportunity-conditioned-floor-actions-v1", "egospeak-opportunity-conditioned-floor-actions-v2",
) )
self.assertTrue((root / "summary_metrics.csv").is_file()) self.assertTrue((root / "summary_metrics.csv").is_file())
self.assertTrue((root / "summary_events.csv").is_file()) self.assertTrue((root / "summary_events.csv").is_file())