Compare commits
5
Commits
7344d0c469
...
5c29457cc6
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5c29457cc6 | ||
|
|
6d0c8b733b | ||
|
|
45facc8bf4 | ||
|
|
d294bd6ede | ||
|
|
d4c900ab13 |
+25
-13
@@ -54,19 +54,32 @@ result.
|
||||
- An opportunity-conditioned dense-output evaluator aligns ENTRY positives and
|
||||
core negatives at non-wearer releases, aligns INT positives and hold-floor
|
||||
negatives at competing non-wearer onsets, and applies strict competing-risk
|
||||
censoring. It uses only the original wearer probabilities, excludes wearer
|
||||
backchannels from headline ENTRY FPR, and records unanchored/ambiguous cases.
|
||||
censoring. ENTRY positives whose selected release is followed by a competing
|
||||
other onset or intervening wearer activity are excluded, and ambiguous
|
||||
multi-competitor INT cases are not assigned an arbitrary onset. It uses only
|
||||
the original wearer probabilities, excludes wearer backchannels from
|
||||
headline ENTRY FPR, and records unanchored/ambiguous cases.
|
||||
|
||||
### Implemented but not GPU-verified
|
||||
|
||||
- MiniCPM-o 4.5 input-only feature extraction has two unmeasured modes: a
|
||||
native 1 Hz state causally held on the 5 Hz grid, and a 5 Hz completed-cache
|
||||
plus temporary-partial-tail branch. The latter refuses the normal
|
||||
head-loading contract unless boundary, replay, snapshot-restoration, and
|
||||
future-audio checks pass. A real-GPU smoke run is still required; see
|
||||
[`docs/minicpmo_streaming_features.md`](docs/minicpmo_streaming_features.md).
|
||||
|
||||
### Measured screening result
|
||||
|
||||
- The released EgoSpeak MiniROADA A+V checkpoint on Sessions 1--3 obtains
|
||||
`74.124` 0.2-second mAP and `61.155` ten-horizon mean mAP. Raw wearer
|
||||
onset/offset anticipation is harder (`24.491/38.024` mean AP). On provisional
|
||||
LLM-assisted matched opportunities, ENTRY AP is `52.487` versus `32.731%`
|
||||
prevalence, while INT/YIELD AP is `20.135`, indistinguishable from its
|
||||
`20.438%` prevalence. Figure-7-style continuous wearer curves therefore
|
||||
support dense forecasting, not semantic floor-control claims. Full curves,
|
||||
counts, exclusions, and caveats are frozen in
|
||||
LLM-assisted matched opportunities, the original v1 evaluator reported
|
||||
ENTRY/INT AP of `52.487/20.135`; those opportunity values are superseded
|
||||
because v1 could cross a competing ENTRY onset and arbitrarily selected among
|
||||
multiple INT competitors. Figure-7-style continuous wearer curves support
|
||||
dense forecasting, not semantic floor-control claims. The unaffected dense
|
||||
curves and historical v1 opportunity record are in
|
||||
[`results/easycom_egospeak_three_level_eval_2026-09-16/`](results/easycom_egospeak_three_level_eval_2026-09-16/README.md).
|
||||
- F1 Frame-set Attention uses fused G+A as one query over the latest half-open
|
||||
causal person set. On the existing three-seed, four-epoch test screen it
|
||||
@@ -74,12 +87,11 @@ result.
|
||||
`64.95099 +/- 0.09451` ten-horizon mean mAP. This is stable, but is not a
|
||||
matched improvement over the historical no-dev pooled result. Its three-seed
|
||||
boundary response/yield AP is `28.423 +/- 0.480` / `48.641 +/- 0.391`; on
|
||||
provisional reviewed opportunities its ENTRY/INT AP is
|
||||
`41.561 +/- 0.581` / `22.991 +/- 0.202`. These values are respectively
|
||||
`-4.461/-1.275` points below the historical pooled-V seed-0 result, so the
|
||||
current F1 screen does not support the hoped-for semantic-selectivity gain.
|
||||
A same-split three-seed pooled control and correct/wrong visual controls
|
||||
remain missing. See the immutable
|
||||
the superseded v1 opportunity evaluator reported ENTRY/INT AP of
|
||||
`41.561 +/- 0.581` / `22.991 +/- 0.202`; these values require a v2 rerun and
|
||||
cannot currently support a semantic-selectivity conclusion. A same-split
|
||||
three-seed pooled control and correct/wrong visual controls remain missing.
|
||||
See the historical
|
||||
[`screening record`](results/easycom_omni_floor_screen_2026-09-16/README.md).
|
||||
|
||||
### Measured local-fusion screens
|
||||
|
||||
@@ -121,6 +121,7 @@ change what the Stage-0 result measures.
|
||||
| `easycom_manifest.py` | Verified on EasyCom | Discover one-minute clips, preserve session timecodes, and check 5 Hz targets |
|
||||
| `qwen_omni_features.py` | Implemented, not executed here | Load only the 3B Thinker and extract fused states from one AV window |
|
||||
| `streaming_turn_model.py` | Verified on EasyCom | Produce one causal Omni state every 200 ms, with causal-prefix and open-stream modes |
|
||||
| `streaming_minicpmo_features_directly.py` | Implemented; real-GPU smoke pending | Extract input-only MiniCPM-o 4.5 native 1 Hz held states or audited 5 Hz cache-plus-partial-tail states |
|
||||
| `TurnPredictionHead` | Trained and evaluated | Map `[2048]` to `[10, 3]` logits |
|
||||
| `train_turn_head.py` | Verified on EasyCom | Train the frozen-Omni anticipation head with the official EasyCom split |
|
||||
| `infer_turn_head.py` | Verified on EasyCom | Save EgoSpeak-format predictions and reproduce per-horizon AP/mAP |
|
||||
|
||||
@@ -10,6 +10,12 @@ head.
|
||||
|
||||
A positive must have a reviewer-selected preceding non-wearer release. Its
|
||||
opportunity starts at that release and resolves at the reviewed wearer entry.
|
||||
If a new non-wearer onset occurs before or exactly at that entry, the release
|
||||
has already been resolved by a competing event and the positive is excluded.
|
||||
The same applies when wearer activity already occurred between the selected
|
||||
release and the reviewed onset: the selected release is then a stale,
|
||||
post-outcome anchor. Both cases are counted separately in
|
||||
`review_diagnostics.skipped`.
|
||||
Core hard negatives start at the same type of non-wearer release and resolve
|
||||
when the same speaker resumes (`ENTRY_OTHER_MIDTURN_PAUSE`) or a different
|
||||
non-wearer takes the next floor (`ENTRY_OTHER_TO_OTHER`). The score is the
|
||||
@@ -27,10 +33,10 @@ the maximum `1 - p_wearer` over 0.2--0.6 seconds, censored strictly at that
|
||||
non-wearer's offset. Reviewed `INT_YIELD` items are positive; other
|
||||
backchannels, failed takeovers, and clear wearer-hold cases are hard negatives.
|
||||
|
||||
Current INT reviews do not select one competing onset. The loader therefore
|
||||
uses the latest supplied other interval that crosses the wearer offset within
|
||||
the annotation tolerance. It records how many intervals were eligible so
|
||||
multi-competitor cases can be audited before the protocol is frozen.
|
||||
Current INT reviews do not select one competing onset. A unique supplied other
|
||||
interval that crosses the wearer offset within the annotation tolerance is
|
||||
used. Cases with multiple eligible intervals are excluded and counted rather
|
||||
than being assigned a deterministic latest-onset anchor.
|
||||
|
||||
## Provisional EgoSpeak run
|
||||
|
||||
@@ -58,21 +64,24 @@ Before manuscript reporting, freeze the 2.0-second ENTRY cap, 0.6-second YIELD
|
||||
cap, positive anchor-selection policy, and multiple-competitor handling using
|
||||
training-side data only.
|
||||
|
||||
## Recorded Sessions 1--3 baseline
|
||||
## Superseded v1 Sessions 1--3 baseline
|
||||
|
||||
The first threshold-free run gives ENTRY AP 52.487% (163 positive, 335 hard
|
||||
negative; 32.731% prevalence) and INT/YIELD AP 20.135% (28 positive, 109 hard
|
||||
negative; 20.438% prevalence). It excludes 81 ENTRY and 5 INT cases as invalid
|
||||
or post-outcome. These are provisional LLM-assisted labels, not human-reviewed
|
||||
confirmatory results. Dense, raw-boundary, and opportunity metrics and their
|
||||
Figure 7 interpretation are frozen together in the
|
||||
The first threshold-free run used the superseded v1 loader and reported ENTRY
|
||||
AP 52.487% (163 positive, 335 hard negative; 32.731% prevalence) and INT/YIELD
|
||||
AP 20.135% (28 positive, 109 hard negative; 20.438% prevalence). It excluded
|
||||
81 ENTRY and 5 INT cases as invalid or post-outcome. These opportunity values
|
||||
must not be reported: v1 allowed a
|
||||
selected ENTRY release to cross an intervening other onset and selected the
|
||||
latest onset in ambiguous multi-competitor INT cases. Dense and raw-boundary
|
||||
metrics are unaffected. Rerun the opportunity evaluation with protocol v2
|
||||
before updating the manuscript. The historical output is retained in the
|
||||
[`2026-09-16 result record`](../results/easycom_egospeak_three_level_eval_2026-09-16/README.md).
|
||||
|
||||
The expanded Omni screen finds no opportunity-ranking gain from the current
|
||||
The expanded v1 Omni screen found no opportunity-ranking gain from the current
|
||||
F1 Frame-set Attention model: its three-seed ENTRY/INT AP is
|
||||
41.561%/22.991%, below the historical pooled-V seed-0 values
|
||||
46.022%/24.266%. Because F1 and that pooled run use different train/dev model-
|
||||
selection protocols, this is a stable negative screen rather than a matched
|
||||
confirmatory comparison. Dense, boundary, per-seed, and provenance details are
|
||||
selection protocols and used the superseded opportunity loader, only its dense
|
||||
and raw-boundary findings remain current. Per-seed and provenance details are
|
||||
in the
|
||||
[`Omni floor screening record`](../results/easycom_omni_floor_screen_2026-09-16/README.md).
|
||||
|
||||
@@ -0,0 +1,204 @@
|
||||
# MiniCPM-o 4.5 causal global features
|
||||
|
||||
`streaming_minicpmo_features_directly.py` uses MiniCPM-o 4.5 only as a frozen
|
||||
audio--visual feature extractor. It never calls `streaming_generate`, never
|
||||
loads TTS, and never lets model-generated speech/listen tokens enter the
|
||||
conversation history.
|
||||
|
||||
## Frozen modes
|
||||
|
||||
### `native-1hz-hold`
|
||||
|
||||
- The model commits one input-only AV chunk per completed second.
|
||||
- The newest completed state is forward-held onto the 5 Hz target grid.
|
||||
- The first four rows (0.2--0.8 s) use an all-zero initialization. The 1.0-s
|
||||
state is never backfilled into those rows.
|
||||
- `state_source_times`, `native_update_mask`, and `initialization_mask` make
|
||||
this alignment auditable.
|
||||
|
||||
### `cached-partial-5hz`
|
||||
|
||||
- Completed one-second chunks remain in the official MiniCPM streaming cache.
|
||||
- At a non-boundary 200-ms cutoff, the extractor snapshots the LLM cache,
|
||||
audio-encoder cache, streaming Mel buffer, chunk index, and RNG state through
|
||||
MiniCPM's official speculative-state API.
|
||||
- It appends only the current half-open AV tail, reads the final normalized
|
||||
Qwen3 hidden state, and restores the snapshot. Temporary closure never enters
|
||||
the main cache.
|
||||
- Native commits retain MiniCPM's normal streaming-window policy. Destructive
|
||||
round pruning is suspended only inside the temporary transaction because the
|
||||
upstream snapshot can truncate appended KV tokens but cannot recreate a
|
||||
previously pruned round.
|
||||
- At a one-second boundary, a complete native chunk is committed.
|
||||
|
||||
The 5 Hz branch is saved as the normal `mode=causal-prefix` contract only after
|
||||
all mandatory checks pass:
|
||||
|
||||
1. a temporary full chunk matches the native committed state at a one-second
|
||||
boundary;
|
||||
2. a clean stream replay matches the cached result;
|
||||
3. changing audio strictly after the cutoff leaves the current state exactly
|
||||
unchanged;
|
||||
4. snapshot restoration leaves LLM/audio cache lengths and checksums, chunk
|
||||
history, position offset, and the audio chunk index unchanged.
|
||||
|
||||
With `--allow-unverified-partial-tail`, a failed or too-short audit may be
|
||||
saved only as `mode=minicpmo-unverified-partial-tail`; the default head loader
|
||||
rejects that artifact.
|
||||
|
||||
## Environment
|
||||
|
||||
The inspected upstream revision requires Python 3.10 and
|
||||
`transformers==4.51.0`. Use a dedicated environment rather than changing the
|
||||
working Qwen environment.
|
||||
|
||||
```bash
|
||||
python -m pip install \
|
||||
"transformers==4.51.0" \
|
||||
accelerate \
|
||||
"torch>=2.3.0,<=2.8.0" \
|
||||
"torchaudio<=2.8.0" \
|
||||
"minicpmo-utils>=1.0.5" \
|
||||
"qwen-omni-utils[decord]"
|
||||
```
|
||||
|
||||
The commands below use the upstream revision inspected during implementation,
|
||||
`1f761131fa83f5ed3cd6f2f22b225c4501d154fa`. If the model is already stored
|
||||
locally, replace `--model-id` with that snapshot directory and record its
|
||||
immutable revision separately.
|
||||
|
||||
## Required smoke sequence
|
||||
|
||||
First validate the source timeline and decoder without loading MiniCPM:
|
||||
|
||||
```bash
|
||||
python streaming_minicpmo_features_directly.py \
|
||||
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
|
||||
--end 3 \
|
||||
--mode native-1hz-hold \
|
||||
--decoder decord \
|
||||
--media-only
|
||||
```
|
||||
|
||||
Then extract three seconds with the native held-state baseline:
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0 python streaming_minicpmo_features_directly.py \
|
||||
--model-id openbmb/MiniCPM-o-4_5 \
|
||||
--revision 1f761131fa83f5ed3cd6f2f22b225c4501d154fa \
|
||||
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
|
||||
--end 3 \
|
||||
--mode native-1hz-hold \
|
||||
--dtype bfloat16 \
|
||||
--attention sdpa \
|
||||
--device cuda:0 \
|
||||
--decoder decord \
|
||||
--decoder-threads 1 \
|
||||
--max-open-readers 1 \
|
||||
--output /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/smoke/Session_4_native_3s.pt
|
||||
```
|
||||
|
||||
Run the verified cached partial-tail smoke test separately:
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0 python streaming_minicpmo_features_directly.py \
|
||||
--model-id openbmb/MiniCPM-o-4_5 \
|
||||
--revision 1f761131fa83f5ed3cd6f2f22b225c4501d154fa \
|
||||
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
|
||||
--end 3 \
|
||||
--mode cached-partial-5hz \
|
||||
--verify-every 1 \
|
||||
--future-intervention-every 1 \
|
||||
--dtype bfloat16 \
|
||||
--attention sdpa \
|
||||
--device cuda:0 \
|
||||
--decoder decord \
|
||||
--decoder-threads 1 \
|
||||
--max-open-readers 1 \
|
||||
--output /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/smoke/Session_4_partial_3s.pt
|
||||
```
|
||||
|
||||
Do not add `--allow-unverified-partial-tail` to a paper run. A verification
|
||||
failure is a protocol failure to diagnose, not a tolerance to bypass.
|
||||
|
||||
Inspect both artifacts before a full session:
|
||||
|
||||
```bash
|
||||
python - <<'PY'
|
||||
from pathlib import Path
|
||||
import torch
|
||||
|
||||
root = Path("/data1/yfzx/wangx/Datasets/easycom/minicpmo_features/smoke")
|
||||
for name in ("Session_4_native_3s.pt", "Session_4_partial_3s.pt"):
|
||||
payload = torch.load(root / name, map_location="cpu", weights_only=False)
|
||||
print(name)
|
||||
print(" mode:", payload["mode"], payload["extractor_mode"])
|
||||
print(" states:", tuple(payload["states"].shape))
|
||||
print(" times:", payload["times"][:6].tolist())
|
||||
print(" source times:", payload["state_source_times"][:6].tolist())
|
||||
print(" verified:", payload["partial_tail_admissible"])
|
||||
print(" checks:", payload["verification"])
|
||||
PY
|
||||
```
|
||||
|
||||
Expected alignment:
|
||||
|
||||
```text
|
||||
native times: 0.2 0.4 0.6 0.8 1.0 1.2 ...
|
||||
native source time: 0.0 0.0 0.0 0.0 1.0 1.0 ...
|
||||
partial source time: 0.2 0.4 0.6 0.8 1.0 1.2 ...
|
||||
```
|
||||
|
||||
## Full Session 4
|
||||
|
||||
Native 1 Hz held features:
|
||||
|
||||
```bash
|
||||
mkdir -p /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/native_1hz_hold
|
||||
|
||||
CUDA_VISIBLE_DEVICES=0 python streaming_minicpmo_features_directly.py \
|
||||
--model-id openbmb/MiniCPM-o-4_5 \
|
||||
--revision 1f761131fa83f5ed3cd6f2f22b225c4501d154fa \
|
||||
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
|
||||
--target /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame/Session_4.npy \
|
||||
--mode native-1hz-hold \
|
||||
--dtype bfloat16 \
|
||||
--attention sdpa \
|
||||
--device cuda:0 \
|
||||
--decoder decord \
|
||||
--decoder-threads 1 \
|
||||
--max-open-readers 1 \
|
||||
--frame-cache-size 32 \
|
||||
--output /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/native_1hz_hold/Session_4.pt
|
||||
```
|
||||
|
||||
Verified 5 Hz cached partial-tail features:
|
||||
|
||||
```bash
|
||||
mkdir -p /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/cached_partial_5hz
|
||||
|
||||
CUDA_VISIBLE_DEVICES=0 python streaming_minicpmo_features_directly.py \
|
||||
--model-id openbmb/MiniCPM-o-4_5 \
|
||||
--revision 1f761131fa83f5ed3cd6f2f22b225c4501d154fa \
|
||||
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
|
||||
--target /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame/Session_4.npy \
|
||||
--mode cached-partial-5hz \
|
||||
--dtype bfloat16 \
|
||||
--attention sdpa \
|
||||
--device cuda:0 \
|
||||
--decoder decord \
|
||||
--decoder-threads 1 \
|
||||
--max-open-readers 1 \
|
||||
--frame-cache-size 32 \
|
||||
--output /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/cached_partial_5hz/Session_4.pt
|
||||
```
|
||||
|
||||
The full run still executes the first clean-replay and future-intervention
|
||||
checks plus every one-second boundary check. Add `--verify-every 250` and
|
||||
`--future-intervention-every 250` for periodic 50-second audits; smaller values
|
||||
make long sessions increasingly expensive because a clean audit replays the
|
||||
prefix from the session start.
|
||||
|
||||
After extraction, either directory can be passed directly as
|
||||
`train_turn_head.py --feature-root ...` because verified artifacts retain the
|
||||
existing dense global-feature contract.
|
||||
@@ -168,12 +168,13 @@ forecasting 的定性解释,不能代替 boundary 或 semantic opportunity 评
|
||||
曲线和限制见
|
||||
[`three-level baseline record`](../results/easycom_egospeak_three_level_eval_2026-09-16/README.md)。
|
||||
|
||||
当前 F1 current-frame person-set screen 的三种子结果同样没有支持 semantic floor
|
||||
selectivity:ENTRY/INT opportunity AP 为 `41.561 +/- 0.581` / `22.991 +/- 0.202`,
|
||||
低于历史 pooled-V seed-0 的 `46.022/24.266`;dense mean mAP 也为
|
||||
`64.951 +/- 0.095`,没有超过历史 pooled-V 的 `65.400`。由于两者 train/dev
|
||||
选择协议不同,这只能判为稳定的负向 screening,正式归因前必须补同协议三种子 pooled
|
||||
control。详见 [`Omni floor screening record`](../results/easycom_omni_floor_screen_2026-09-16/README.md)。
|
||||
当前 F1 current-frame person-set screen 的 dense mean mAP 为
|
||||
`64.951 +/- 0.095`,没有超过历史 pooled-V 的 `65.400`。原 v1 evaluator 给出的
|
||||
ENTRY/INT opportunity AP(`41.561 +/- 0.581` / `22.991 +/- 0.202`)已失效:v1
|
||||
可能跨过 competing ENTRY onset,且会在多个 INT competitor 中任意选择 latest onset。
|
||||
必须用 v2 重跑后才能判断 semantic floor selectivity;正式归因还必须补同协议三种子
|
||||
pooled control。历史数值与勘误见
|
||||
[`Omni floor screening record`](../results/easycom_omni_floor_screen_2026-09-16/README.md)。
|
||||
|
||||
### P3:第一优先方法——匿名逐人时序旁路
|
||||
|
||||
|
||||
@@ -8,14 +8,18 @@ event head.
|
||||
|
||||
ENTRY opportunities are anchored at a reviewed preceding non-wearer release.
|
||||
Positive opportunities resolve at the wearer entry; core hard negatives
|
||||
resolve when the same or a different non-wearer speaks next. The score is the
|
||||
maximum wearer probability over available 0.2--2.0 s forecasts up to that
|
||||
competing resolution.
|
||||
resolve when the same or a different non-wearer speaks next. A positive is
|
||||
excluded if another non-wearer onset competes before (or exactly at) its wearer
|
||||
entry, or if wearer activity already occurred after the selected release. The
|
||||
score is the maximum wearer probability over available 0.2--2.0 s forecasts up
|
||||
to the uncontested resolution.
|
||||
|
||||
YIELD opportunities are anchored at a non-wearer onset during wearer speech.
|
||||
The score is the maximum future wearer-inactivity probability over 0.2--0.6 s,
|
||||
censored at that non-wearer's offset. Positive and hard-negative examples
|
||||
therefore begin at the same kind of observed conversational opportunity.
|
||||
censored at that non-wearer's offset. Reviews with multiple eligible
|
||||
competing onsets are excluded until the annotation selects one. Positive and
|
||||
hard-negative examples therefore begin at the same kind of unambiguous
|
||||
observed conversational opportunity.
|
||||
|
||||
Prediction arrays use the repository contract
|
||||
``[target_time, class, future_horizon]``. All scores are read diagonally from
|
||||
@@ -178,17 +182,83 @@ def _candidate_region(
|
||||
return start, end
|
||||
|
||||
|
||||
def _int_competing_interval(
|
||||
def _entry_selected_option(
|
||||
candidate: dict[str, Any],
|
||||
review: dict[str, Any],
|
||||
*,
|
||||
selected_release_seconds: float,
|
||||
uid: str,
|
||||
) -> dict[str, Any] | None:
|
||||
evidence = candidate.get("temporal_evidence")
|
||||
evidence = evidence if isinstance(evidence, dict) else {}
|
||||
options = evidence.get("opportunity_options")
|
||||
if not isinstance(options, list):
|
||||
return None
|
||||
selected_uid = review.get("selected_opportunity_uid")
|
||||
matches: list[dict[str, Any]] = []
|
||||
for index, option in enumerate(options):
|
||||
if not isinstance(option, dict):
|
||||
continue
|
||||
option_seconds = _optional_seconds(
|
||||
option.get("anchor_seconds"),
|
||||
field=f"opportunity_options[{index}].anchor_seconds",
|
||||
uid=uid,
|
||||
)
|
||||
if selected_uid is not None and option.get("uid") == selected_uid:
|
||||
matches.append(option)
|
||||
elif selected_uid is None and option_seconds is not None and math.isclose(
|
||||
option_seconds, selected_release_seconds, abs_tol=1e-7
|
||||
):
|
||||
matches.append(option)
|
||||
if len(matches) > 1:
|
||||
raise ValueError(f"{uid}: selected ENTRY opportunity is not unique")
|
||||
return matches[0] if matches else None
|
||||
|
||||
|
||||
def _entry_competing_onsets(
|
||||
candidate: dict[str, Any],
|
||||
*,
|
||||
selected_release_seconds: float,
|
||||
behavior_anchor_seconds: float,
|
||||
uid: str,
|
||||
) -> list[float]:
|
||||
"""Return other onsets that resolve a release before the wearer entry."""
|
||||
|
||||
evidence = candidate.get("temporal_evidence")
|
||||
evidence = evidence if isinstance(evidence, dict) else {}
|
||||
raw_intervals = evidence.get("gap_other_intervals")
|
||||
if not isinstance(raw_intervals, list):
|
||||
raw_intervals = []
|
||||
onsets: set[float] = set()
|
||||
for index, interval in enumerate(raw_intervals):
|
||||
if not isinstance(interval, dict):
|
||||
continue
|
||||
onset = _optional_seconds(
|
||||
interval.get("onset_seconds"),
|
||||
field=f"gap_other_intervals[{index}].onset_seconds",
|
||||
uid=uid,
|
||||
)
|
||||
if onset is None:
|
||||
continue
|
||||
if (
|
||||
onset >= selected_release_seconds - 1e-7
|
||||
and onset <= behavior_anchor_seconds + 1e-7
|
||||
):
|
||||
onsets.add(onset)
|
||||
return sorted(onsets)
|
||||
|
||||
|
||||
def _int_competing_intervals(
|
||||
candidate: dict[str, Any],
|
||||
*,
|
||||
behavior_anchor_seconds: float,
|
||||
uid: str,
|
||||
) -> tuple[float, float, int] | None:
|
||||
) -> list[tuple[float, float]]:
|
||||
evidence = candidate.get("temporal_evidence")
|
||||
evidence = evidence if isinstance(evidence, dict) else {}
|
||||
raw_intervals = evidence.get("nearby_other_intervals")
|
||||
if not isinstance(raw_intervals, list):
|
||||
return None
|
||||
return []
|
||||
eligible: list[tuple[float, float]] = []
|
||||
for index, interval in enumerate(raw_intervals):
|
||||
if not isinstance(interval, dict):
|
||||
@@ -209,13 +279,7 @@ def _int_competing_interval(
|
||||
continue
|
||||
if onset < behavior_anchor_seconds and offset > onset:
|
||||
eligible.append((onset, offset))
|
||||
if not eligible:
|
||||
return None
|
||||
# A semantic INT review does not currently select one competing onset.
|
||||
# The latest interval crossing the wearer offset is the deterministic,
|
||||
# most proximal causal candidate. Multiplicity is surfaced for audit.
|
||||
onset, offset = max(eligible, key=lambda item: (item[0], item[1]))
|
||||
return onset, offset, len(eligible)
|
||||
return sorted(set(eligible))
|
||||
|
||||
|
||||
def load_opportunity_events(
|
||||
@@ -278,6 +342,38 @@ def load_opportunity_events(
|
||||
if selected_release is None:
|
||||
skipped["entry_positive_without_selected_release"] += 1
|
||||
continue
|
||||
selected_option = _entry_selected_option(
|
||||
candidate,
|
||||
review,
|
||||
selected_release_seconds=selected_release,
|
||||
uid=uid,
|
||||
)
|
||||
if (
|
||||
selected_option is not None
|
||||
and selected_option.get(
|
||||
"wearer_active_between_release_and_onset"
|
||||
) is True
|
||||
):
|
||||
skipped["entry_positive_post_outcome_wearer_activity"] += 1
|
||||
continue
|
||||
competing_onsets = _entry_competing_onsets(
|
||||
candidate,
|
||||
selected_release_seconds=selected_release,
|
||||
behavior_anchor_seconds=behavior,
|
||||
uid=uid,
|
||||
)
|
||||
option_reports_competitor = (
|
||||
selected_option is not None
|
||||
and selected_option.get(
|
||||
"other_onset_between_release_and_onset"
|
||||
) is True
|
||||
)
|
||||
if option_reports_competitor and not competing_onsets:
|
||||
skipped["entry_positive_competing_onset_unresolved"] += 1
|
||||
continue
|
||||
if competing_onsets:
|
||||
skipped["entry_positive_competed_before_behavior"] += 1
|
||||
continue
|
||||
anchor = selected_release
|
||||
resolution = behavior
|
||||
censor = behavior
|
||||
@@ -298,17 +394,21 @@ def load_opportunity_events(
|
||||
competing_intervals = 0
|
||||
else:
|
||||
if positive:
|
||||
interval = _int_competing_interval(
|
||||
intervals = _int_competing_intervals(
|
||||
candidate,
|
||||
behavior_anchor_seconds=behavior,
|
||||
uid=uid,
|
||||
)
|
||||
if interval is None:
|
||||
if not intervals:
|
||||
skipped["int_positive_without_crossing_other"] += 1
|
||||
continue
|
||||
anchor, censor, competing_intervals = interval
|
||||
if len(intervals) > 1:
|
||||
skipped["int_positive_ambiguous_competing_others"] += 1
|
||||
continue
|
||||
anchor, censor = intervals[0]
|
||||
competing_intervals = 1
|
||||
resolution = behavior
|
||||
basis = "latest_other_onset_crossing_wearer_offset"
|
||||
basis = "unique_other_onset_crossing_wearer_offset"
|
||||
elif subtype_value in INT_HARD_NEGATIVES:
|
||||
if source in INT_MINED_SOURCES:
|
||||
anchor, censor = _candidate_region(candidate, uid=uid)
|
||||
@@ -316,17 +416,21 @@ def load_opportunity_events(
|
||||
competing_intervals = 1
|
||||
basis = "mined_competing_other_onset"
|
||||
else:
|
||||
interval = _int_competing_interval(
|
||||
intervals = _int_competing_intervals(
|
||||
candidate,
|
||||
behavior_anchor_seconds=behavior,
|
||||
uid=uid,
|
||||
)
|
||||
if interval is None:
|
||||
if not intervals:
|
||||
skipped["int_hn_without_competing_other"] += 1
|
||||
continue
|
||||
anchor, censor, competing_intervals = interval
|
||||
if len(intervals) > 1:
|
||||
skipped["int_hn_ambiguous_competing_others"] += 1
|
||||
continue
|
||||
anchor, censor = intervals[0]
|
||||
competing_intervals = 1
|
||||
resolution = behavior
|
||||
basis = "latest_other_onset_crossing_non_yield_offset"
|
||||
basis = "unique_other_onset_crossing_non_yield_offset"
|
||||
else:
|
||||
skipped["int_unsupported_hard_negative"] += 1
|
||||
continue
|
||||
@@ -731,8 +835,8 @@ def main() -> None:
|
||||
else calibration_diagnostics["num_provisional_llm"]
|
||||
)
|
||||
payload = {
|
||||
"schema_version": 1,
|
||||
"protocol": "egospeak-opportunity-conditioned-floor-actions-v1",
|
||||
"schema_version": 2,
|
||||
"protocol": "egospeak-opportunity-conditioned-floor-actions-v2",
|
||||
"semantic_scope": (
|
||||
"LLM-assisted opportunity labels with dense wearer-activity scores; "
|
||||
"this is provisional and no human review is implied."
|
||||
@@ -757,6 +861,13 @@ def main() -> None:
|
||||
"yield_max_seconds": yield_max,
|
||||
"entry_headline_hard_negatives": sorted(ENTRY_CORE_HARD_NEGATIVES),
|
||||
"entry_backchannel_policy": "separate diagnostic; excluded from headline",
|
||||
"entry_competing_risk_policy": (
|
||||
"exclude selected-release positives with an other onset at or before "
|
||||
"the wearer entry, or with intervening wearer activity"
|
||||
),
|
||||
"int_multiple_competitor_policy": (
|
||||
"exclude until one competing onset is selected by review"
|
||||
),
|
||||
"matched_recalls": recalls,
|
||||
"review_diagnostics": review_diagnostics,
|
||||
"calibration_review_diagnostics": calibration_diagnostics,
|
||||
|
||||
+86
-41
@@ -339,9 +339,12 @@ non-wearer release. It resolves positively when the wearer enters, or
|
||||
negatively when the same non-wearer resumes or a different non-wearer takes the
|
||||
next floor. From the first causal 5-Hz state that observes the release, the
|
||||
entry score is the maximum projected wearer probability before that competing
|
||||
resolution, capped at 2.0~s. Positive entries without a reviewer-selected
|
||||
release are reported as unanchored and excluded from this matched-opportunity
|
||||
analysis; they remain available to the separate raw-onset analysis.
|
||||
resolution, capped at 2.0~s. A selected-release positive is excluded if another
|
||||
non-wearer onset occurs before or exactly at the wearer entry, or if wearer
|
||||
activity already occurred after the selected release. Positive entries without
|
||||
a reviewer-selected release are reported as unanchored and excluded from this
|
||||
matched-opportunity analysis; they remain available to the separate raw-onset
|
||||
analysis.
|
||||
|
||||
An INT opportunity begins when a non-wearer starts while the wearer holds the
|
||||
floor. It resolves positively when the reviewed behavior is an INT-yield, and
|
||||
@@ -352,8 +355,9 @@ strictly at the competing non-wearer offset. All prediction values are read
|
||||
diagonally from one causal source row of the target-aligned forecast tensor.
|
||||
No input after the observed action outcome is used. For current INT-positive
|
||||
annotations, which do not select a competing onset explicitly, the
|
||||
implementation uses the latest supplied non-wearer interval crossing the
|
||||
wearer offset and surfaces cases with multiple eligible intervals for audit.
|
||||
implementation evaluates only a unique supplied non-wearer interval crossing
|
||||
the wearer offset; cases with multiple eligible intervals are excluded and
|
||||
counted for audit rather than assigned an arbitrary onset.
|
||||
|
||||
The headline entry hard negatives are same-speaker mid-turn pauses and other-
|
||||
to-other transfers. Wearer backchannels are not pooled into headline false-
|
||||
@@ -394,46 +398,60 @@ latency claim.
|
||||
|
||||
\subsection{Main Comparison}
|
||||
|
||||
Table~\ref{tab:main} compares the proposed causal local audiovisual adapter
|
||||
with an EgoSpeak reproduction, the frozen Qwen2.5-Omni-3B global state $G$,
|
||||
matched PEFT, $G$ plus recent local audio, and a parameter- and history-matched
|
||||
temporal local branch. Qwen2.5-Omni-7B is included only as a capacity-scaling
|
||||
control. All methods use the same split, causal inputs, targets, and evaluation
|
||||
protocol. Event thresholds are selected on training-side calibration data and
|
||||
frozen before test evaluation. \todo{Freeze all baseline implementations and
|
||||
verify identical targets, causal validity, and evaluation.}
|
||||
Table~\ref{tab:main} places the released EgoSpeak checkpoint beside the
|
||||
currently available Omni screening runs and reserves a row for the final causal
|
||||
local audiovisual model. Historical unmatched runs are identified explicitly
|
||||
and cannot support confirmatory attribution. The final comparison will use the
|
||||
same split, causal inputs, targets, training/selection protocol, and three
|
||||
random seeds; event thresholds will be selected on training-side calibration
|
||||
data and frozen before test evaluation. \todo{Freeze all baseline
|
||||
implementations and verify identical targets, causal validity, and evaluation.}
|
||||
|
||||
\begin{table}[t]
|
||||
\caption{Confirmatory EasyCom results. Entry and INT false-positive rates use
|
||||
the opportunity-conditioned activity score at matched recall $R=0.8$; E/I AP
|
||||
is the corresponding opportunity AP. $\dagger$ denotes provisional
|
||||
LLM-assisted labels pending human review.}
|
||||
\begin{table*}[t]
|
||||
\caption{EasyCom Sessions~1--3 dense future-activity and raw-boundary
|
||||
screening. All values are percentages. ``Avg.'' averages dense mAP over the ten
|
||||
horizons; On/Off are mean raw wearer onset/offset AP and are timing diagnostics,
|
||||
not semantic ENTRY/INT scores. $\dagger$ denotes historical seed-0 runs with an
|
||||
unmatched training/selection protocol; $\ddagger$ denotes a three-seed screen.
|
||||
Missing horizon curves and the final matched comparison will be filled after
|
||||
the matched dense/boundary reruns.}
|
||||
\label{tab:main}
|
||||
\centering
|
||||
\small
|
||||
\resizebox{\columnwidth}{!}{%
|
||||
\begin{tabular}{lccccc}
|
||||
\scriptsize
|
||||
\setlength{\tabcolsep}{2.6pt}
|
||||
\resizebox{\textwidth}{!}{%
|
||||
\begin{tabular}{@{}lccccccccccccc@{}}
|
||||
\hline
|
||||
Method & 0.2-s mAP $\uparrow$ & Mean mAP $\uparrow$
|
||||
& Entry FPR@$R=.8$ $\downarrow$ & INT FPR@$R=.8$ $\downarrow$ & E/I AP $\uparrow$\\
|
||||
& \multicolumn{10}{c}{Dense mAP at prediction horizon (s) $\uparrow$}
|
||||
& & \multicolumn{2}{c}{Raw boundary AP $\uparrow$}\\
|
||||
\cline{2-11}\cline{13-14}
|
||||
Method & .2 & .4 & .6 & .8 & 1.0 & 1.2 & 1.4 & 1.6 & 1.8 & 2.0
|
||||
& Avg. & On & Off\\
|
||||
\hline
|
||||
EgoSpeak MiniROADA A+V & 74.12 & 61.16 & \todo{--} & \todo{--} & $52.49/20.13^\dagger$\\
|
||||
Omni-3B frozen $G$ & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
|
||||
Omni-3B matched PEFT & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
|
||||
$G+$local audio & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
|
||||
$G+$matched temporal local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
|
||||
$G+$causal local AV (ours) & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
|
||||
Omni-7B scaling control & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
|
||||
EgoSpeak MiniROADA A+V
|
||||
& 74.20 & 68.35 & 64.31 & 61.13 & 59.77 & 58.81 & 57.50 & 57.09 & 55.95 & 54.82
|
||||
& 61.19 & 24.49 & 38.02\\
|
||||
Omni $G{+}A^{\dagger}$
|
||||
& 81.72 & -- & -- & -- & -- & -- & -- & -- & -- & --
|
||||
& 65.11 & 25.96 & 45.25\\
|
||||
Omni $G{+}A{+}$pooled-$V^{\dagger}$
|
||||
& 82.03 & -- & -- & -- & -- & -- & -- & -- & -- & --
|
||||
& 65.40 & 26.02 & 45.37\\
|
||||
Current-frame F1$^{\ddagger}$
|
||||
& $81.78{\pm}0.16$ & -- & -- & -- & -- & -- & -- & -- & -- & --
|
||||
& $64.95{\pm}0.09$ & -- & --\\
|
||||
$G+$causal local AV (ours)
|
||||
& -- & -- & -- & -- & -- & -- & -- & -- & -- & --
|
||||
& -- & -- & --\\
|
||||
\hline
|
||||
\end{tabular}}
|
||||
\end{table}
|
||||
\end{table*}
|
||||
|
||||
\textbf{Dense forecasting.}
|
||||
The EgoSpeak baseline reaches 74.12\% 0.2-s mAP and 61.16\% mean mAP; its raw
|
||||
wearer-onset/offset mean APs are 24.49/38.02\%. On provisional reviewed
|
||||
opportunities, ENTRY AP is 52.49\% versus 32.73\% prevalence, whereas INT AP is
|
||||
20.13\% versus 20.44\%; continuous wearer-score plots therefore do not alone
|
||||
establish semantic floor-yield selectivity.
|
||||
The EgoSpeak baseline reaches 74.20\% 0.2-s mAP and 61.19\% mean mAP; its raw
|
||||
wearer-onset/offset mean APs are 24.49/38.02\%. Opportunity-conditioned values
|
||||
remain pending a protocol-v2 rerun and human review; continuous wearer-score
|
||||
plots do not alone establish semantic floor-yield selectivity.
|
||||
The proposed local AV adapter improves 0.2-s mAP from
|
||||
\todo{$X$ to $Y$} over the strongest matched baseline and improves
|
||||
mean mAP by \todo{$Z$ points}.
|
||||
@@ -460,6 +478,34 @@ beyond additional temporal capacity}.
|
||||
\todo{If this difference is not statistically supported, state instead that
|
||||
the local temporal branch accounts for most of the gain.}
|
||||
|
||||
\subsection{Qualitative Floor-Event Analysis}
|
||||
|
||||
We will compare EgoSpeak with the final matched model on human-confirmed
|
||||
examples selected by pre-declared event subtype rather than post-hoc score
|
||||
extremes. Each case will align participant-resolved activity, minimal transcript
|
||||
context, the opportunity anchor and competing-risk censor, and the two models'
|
||||
causally emitted wearer forecasts. The comparison will include a genuine wearer
|
||||
entry, an other-to-other false-entry opportunity, and an INT-yield paired with a
|
||||
non-takeover overlap; a shared failure case will be retained when space permits.
|
||||
|
||||
\begin{figure*}[t]
|
||||
\centering
|
||||
\fbox{\parbox[c][0.115\textheight][c]{0.965\textwidth}{
|
||||
\centering
|
||||
\textbf{Qualitative comparison placeholder}\\[0.7em]
|
||||
\begin{tabular}{ccc}
|
||||
Case A: wearer entry & Case B: other$\rightarrow$other HN
|
||||
& Case C: INT-yield / failed takeover\\
|
||||
participant timeline + transcript & opportunity anchor + censor
|
||||
& EgoSpeak vs. proposed forecast\\
|
||||
\end{tabular}
|
||||
}}
|
||||
\caption{Planned qualitative floor-event comparison. Final panels will use
|
||||
human-confirmed protocol-v2 cases and show the same causal source time for both
|
||||
models; the figure is a layout placeholder and contains no result claim.}
|
||||
\label{fig:qualitative_floor_events}
|
||||
\end{figure*}
|
||||
|
||||
\subsection{Information Opportunity}
|
||||
|
||||
Before designing the deployable adapter, we measured whether local floor
|
||||
@@ -570,7 +616,7 @@ future-media intervention test. Report within-backbone 5-Hz versus 1-Hz
|
||||
deltas plus cache/no-cache throughput and memory; do not interpret unmatched
|
||||
Qwen--MiniCPM raw scores as a pure backbone effect.}
|
||||
|
||||
\subsection{Limitations and Qualitative Analysis}
|
||||
\subsection{Limitations}
|
||||
|
||||
The results support \todo{precise supported claim}, but several limitations
|
||||
remain.
|
||||
@@ -580,10 +626,9 @@ tracking assumption}.
|
||||
Third, \todo{failure condition: occlusion, overlapping speakers, rapid floor
|
||||
exchange, etc.} remains challenging.
|
||||
|
||||
\todo{Add one compact, pre-selected qualitative timeline only if space permits:
|
||||
one multi-other false-entry improvement paired with one preserved real wearer
|
||||
onset or one failure. Examples illustrate aggregate rates and never establish
|
||||
the claim by themselves.}
|
||||
\todo{Replace Fig.~\ref{fig:qualitative_floor_events} with compact, pre-selected
|
||||
human-confirmed timelines. Examples illustrate aggregate rates and never
|
||||
establish the claim by themselves.}
|
||||
|
||||
\section{Conclusion}
|
||||
\label{sec:conclusion}
|
||||
|
||||
@@ -1,5 +1,11 @@
|
||||
# EasyCom EgoSpeak three-level evaluation
|
||||
|
||||
> **Protocol erratum.** The dense and raw-boundary results in this record remain
|
||||
> valid. The opportunity-conditioned ENTRY/INT values were produced by the
|
||||
> superseded v1 loader, which could cross an intervening ENTRY competitor and
|
||||
> selected the latest onset in ambiguous multi-competitor INT cases. Do not cite
|
||||
> those opportunity values; rerun protocol v2 and create a new result record.
|
||||
|
||||
This record freezes the first Sessions 1--3 evaluation of the released
|
||||
EgoSpeak MiniROADA audio-visual checkpoint under three deliberately separate
|
||||
questions. The values below were produced on the remote experiment host; the
|
||||
@@ -15,7 +21,7 @@ prediction and per-event artifacts remain on that host and are not committed.
|
||||
- Raw boundary anticipation: threshold-free AP for wearer `OFF->ON` and
|
||||
`ON->OFF` changes. These are timing-derived diagnostics, not semantic EOT or
|
||||
INT claims.
|
||||
- Opportunity-conditioned behavior: one causally censored activity-derived
|
||||
- Opportunity-conditioned behavior: historical v1 activity-derived
|
||||
score per reviewed ENTRY or INT/YIELD opportunity. Labels are provisional
|
||||
two-pass LLM-assisted reviews and have not been human reviewed.
|
||||
- No test-set threshold is fitted for any AP reported here. Session 4 is needed
|
||||
@@ -34,15 +40,16 @@ mean AP, respectively. These lower values do not contradict the dense result:
|
||||
dense AP rewards correct future state over sustained regions, whereas a raw
|
||||
boundary event is counted once and has many harder non-boundary comparisons.
|
||||
|
||||
On the provisional opportunity labels, ENTRY AP is **52.487%** for 163
|
||||
The superseded v1 run reported ENTRY AP of **52.487%** for 163
|
||||
positives and 335 hard negatives, versus 32.731% positive prevalence
|
||||
(+19.756 points; 1.60x prevalence). INT/YIELD AP is **20.135%** for 28 positives
|
||||
and 109 hard negatives, essentially equal to its 20.438% prevalence. This is
|
||||
consistent with the output contract: wearer probability can rank many entry
|
||||
opportunities, but `1-p(wearer)` does not encode whether an interruption should
|
||||
cause a yield. The ENTRY and INT loaders additionally exclude 81 and 5
|
||||
invalid/post-outcome cases; these exclusions require an audit before any final
|
||||
paper claim.
|
||||
and 109 hard negatives, essentially equal to its 20.438% prevalence. These
|
||||
values are retained only for provenance and must be replaced by a v2 rerun. The
|
||||
original pattern was consistent with the output contract: wearer probability
|
||||
can rank many entry opportunities, but `1-p(wearer)` does not encode whether an
|
||||
interruption should cause a yield. The ENTRY and INT loaders additionally
|
||||
exclude 81 and 5 invalid/post-outcome cases; these exclusions require an audit
|
||||
before any final paper claim.
|
||||
|
||||
## Relation to EgoSpeak Figure 7
|
||||
|
||||
@@ -73,4 +80,3 @@ Machine-readable curves and counts are in [`summary.json`](summary.json) and
|
||||
the compact comparison is in [`metrics.csv`](metrics.csv). The opportunity
|
||||
protocol and reproduction command are documented in
|
||||
[`docs/easycom_opportunity_event_evaluation.md`](../../docs/easycom_opportunity_event_evaluation.md).
|
||||
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
{
|
||||
"recorded_at": "2026-09-16",
|
||||
"source": "user-provided remote experiment output",
|
||||
"opportunity_protocol_status": "superseded_v1_rerun_required",
|
||||
"opportunity_protocol_erratum": "Dense and raw-boundary metrics remain valid; opportunity metrics must be rerun with egospeak-opportunity-conditioned-floor-actions-v2.",
|
||||
"dataset": "EasyCom",
|
||||
"sessions": ["Session_1", "Session_2", "Session_3"],
|
||||
"model": "EgoSpeak MiniROADA audio-visual released checkpoint",
|
||||
@@ -57,4 +59,3 @@
|
||||
"threshold_free": true,
|
||||
"session4_required_for_reported_metrics": false
|
||||
}
|
||||
|
||||
|
||||
@@ -1,5 +1,12 @@
|
||||
# EasyCom Omni floor-event screening
|
||||
|
||||
> **Protocol erratum.** Dense and raw-boundary metrics remain valid. All
|
||||
> opportunity-conditioned ENTRY/INT values below used the superseded v1 loader,
|
||||
> which could cross an intervening ENTRY competitor and selected the latest
|
||||
> onset in ambiguous multi-competitor INT cases. Retain them for provenance
|
||||
> only; do not use them for model selection or manuscript claims before a v2
|
||||
> rerun.
|
||||
|
||||
This record freezes the first comparison of EgoSpeak, Omni `G+A`, Omni
|
||||
`G+A+pooled-V`, and the three-seed current-frame F1 Frame-set Attention screen
|
||||
on EasyCom Sessions 1--3. Values were produced on the remote experiment host
|
||||
@@ -11,9 +18,8 @@ CSVs remain on that host and are not committed here.
|
||||
1. Dense future activity uses causal-valid framewise AP/mAP over ten horizons.
|
||||
2. Raw wearer endpoint and aggregate-other-boundary AP are threshold-free
|
||||
timing diagnostics; they are not semantic EOT/INT labels.
|
||||
3. Opportunity-conditioned ENTRY and INT/YIELD AP assigns one causally
|
||||
censored activity-derived score to each provisional LLM-assisted reviewed
|
||||
opportunity.
|
||||
3. Opportunity-conditioned ENTRY and INT/YIELD AP is a historical v1 screen
|
||||
that requires rerunning with the corrected v2 competing-risk protocol.
|
||||
|
||||
All opportunity runs use identical denominators: 163 ENTRY positives and 335
|
||||
hard negatives, with 81 invalid/post-outcome cases; and 28 INT positives and
|
||||
@@ -39,7 +45,7 @@ from EgoSpeak to G+A to G+A+pooled-V as follows: onset AP
|
||||
offset/yield-timing gain comes from G+A; pooled visual input adds almost
|
||||
nothing to raw transition timing.
|
||||
|
||||
## Interpretation
|
||||
## Historical v1 interpretation
|
||||
|
||||
- G+A strongly improves dense forecasting and generic wearer-offset/yield
|
||||
timing over EgoSpeak, but not reviewed semantic opportunities. In particular,
|
||||
@@ -77,4 +83,3 @@ nothing to raw transition timing.
|
||||
Exact aggregate values are stored in [`summary.json`](summary.json) and
|
||||
[`metrics.csv`](metrics.csv). The earlier EgoSpeak-only record is
|
||||
[`easycom_egospeak_three_level_eval_2026-09-16`](../easycom_egospeak_three_level_eval_2026-09-16/README.md).
|
||||
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
{
|
||||
"recorded_at": "2026-09-16",
|
||||
"source": "user-provided remote experiment output",
|
||||
"opportunity_protocol_status": "superseded_v1_rerun_required",
|
||||
"opportunity_protocol_erratum": "Dense and raw-boundary metrics remain valid; opportunity metrics must be rerun with egospeak-opportunity-conditioned-floor-actions-v2.",
|
||||
"dataset": "EasyCom",
|
||||
"test_sessions": ["Session_1", "Session_2", "Session_3"],
|
||||
"scale": "percent",
|
||||
@@ -82,4 +84,3 @@
|
||||
"threshold_free_metrics_only": true,
|
||||
"matched_pooled_control_available": false
|
||||
}
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -30,7 +30,10 @@ def reviewed_row(
|
||||
region_start: float,
|
||||
region_end: float,
|
||||
selected_release: float | None = None,
|
||||
selected_release_uid: str | None = None,
|
||||
nearby: list[dict[str, object]] | None = None,
|
||||
opportunity_options: list[dict[str, object]] | None = None,
|
||||
gap_other_intervals: list[dict[str, object]] | None = None,
|
||||
candidate_source: str | None = None,
|
||||
) -> dict[str, object]:
|
||||
event_type = None
|
||||
@@ -45,6 +48,7 @@ def reviewed_row(
|
||||
"human_verdict": verdict,
|
||||
"human_event_type": event_type,
|
||||
"human_subtype": subtype,
|
||||
"selected_opportunity_uid": selected_release_uid,
|
||||
"selected_opportunity_seconds": selected_release,
|
||||
"boundary_correction_seconds": 0,
|
||||
},
|
||||
@@ -59,6 +63,8 @@ def reviewed_row(
|
||||
"behavior_anchor_seconds": anchor,
|
||||
"wearer_offset_seconds": anchor,
|
||||
"nearby_other_intervals": nearby or [],
|
||||
"opportunity_options": opportunity_options or [],
|
||||
"gap_other_intervals": gap_other_intervals or [],
|
||||
},
|
||||
},
|
||||
}
|
||||
@@ -119,7 +125,7 @@ class OpportunityEventTests(unittest.TestCase):
|
||||
diagnostics["skipped"]["entry_backchannel_separate_diagnostic"], 1
|
||||
)
|
||||
|
||||
def test_loader_selects_latest_crossing_other_for_int(self) -> None:
|
||||
def test_loader_excludes_ambiguous_crossing_others_for_int(self) -> None:
|
||||
nearby = [
|
||||
{
|
||||
"onset_seconds": 1.0,
|
||||
@@ -145,11 +151,91 @@ class OpportunityEventTests(unittest.TestCase):
|
||||
)
|
||||
])
|
||||
self.addCleanup(temporary.cleanup)
|
||||
events, _ = load_opportunity_events([path], ["Session_4"])
|
||||
self.assertEqual(len(events), 1)
|
||||
self.assertEqual(events[0].opportunity_anchor_seconds, 2.0)
|
||||
self.assertEqual(events[0].censor_seconds, 3.8)
|
||||
self.assertEqual(events[0].competing_intervals, 2)
|
||||
events, diagnostics = load_opportunity_events([path], ["Session_4"])
|
||||
self.assertEqual(events, [])
|
||||
self.assertEqual(
|
||||
diagnostics["skipped"]["int_positive_ambiguous_competing_others"],
|
||||
1,
|
||||
)
|
||||
|
||||
def test_loader_excludes_entry_positive_at_competing_other_onset(self) -> None:
|
||||
rows = [
|
||||
reviewed_row(
|
||||
"entry-competed",
|
||||
family="entry",
|
||||
verdict="POSITIVE",
|
||||
subtype=None,
|
||||
anchor=1.6,
|
||||
region_start=0.0,
|
||||
region_end=3.0,
|
||||
selected_release=1.0,
|
||||
selected_release_uid="release",
|
||||
opportunity_options=[
|
||||
{
|
||||
"uid": "release",
|
||||
"anchor_seconds": 1.0,
|
||||
"wearer_active_between_release_and_onset": False,
|
||||
"other_onset_between_release_and_onset": True,
|
||||
}
|
||||
],
|
||||
gap_other_intervals=[
|
||||
{"uid": "competitor", "onset_seconds": 1.4}
|
||||
],
|
||||
),
|
||||
reviewed_row(
|
||||
"entry-simultaneous",
|
||||
family="entry",
|
||||
verdict="POSITIVE",
|
||||
subtype=None,
|
||||
anchor=4.0,
|
||||
region_start=3.0,
|
||||
region_end=5.0,
|
||||
selected_release=3.2,
|
||||
gap_other_intervals=[
|
||||
{"uid": "competitor", "onset_seconds": 4.0}
|
||||
],
|
||||
),
|
||||
]
|
||||
temporary, path = self.write_reviews(rows)
|
||||
self.addCleanup(temporary.cleanup)
|
||||
events, diagnostics = load_opportunity_events([path], ["Session_4"])
|
||||
self.assertEqual(events, [])
|
||||
self.assertEqual(
|
||||
diagnostics["skipped"]["entry_positive_competed_before_behavior"],
|
||||
2,
|
||||
)
|
||||
|
||||
def test_loader_excludes_entry_positive_after_prior_wearer_activity(self) -> None:
|
||||
temporary, path = self.write_reviews([
|
||||
reviewed_row(
|
||||
"entry-post-outcome",
|
||||
family="entry",
|
||||
verdict="POSITIVE",
|
||||
subtype=None,
|
||||
anchor=2.0,
|
||||
region_start=0.0,
|
||||
region_end=3.0,
|
||||
selected_release=1.0,
|
||||
selected_release_uid="release",
|
||||
opportunity_options=[
|
||||
{
|
||||
"uid": "release",
|
||||
"anchor_seconds": 1.0,
|
||||
"wearer_active_between_release_and_onset": True,
|
||||
"other_onset_between_release_and_onset": False,
|
||||
}
|
||||
],
|
||||
)
|
||||
])
|
||||
self.addCleanup(temporary.cleanup)
|
||||
events, diagnostics = load_opportunity_events([path], ["Session_4"])
|
||||
self.assertEqual(events, [])
|
||||
self.assertEqual(
|
||||
diagnostics["skipped"][
|
||||
"entry_positive_post_outcome_wearer_activity"
|
||||
],
|
||||
1,
|
||||
)
|
||||
|
||||
def test_entry_hn_requires_matching_independently_mined_source(self) -> None:
|
||||
rows = [
|
||||
@@ -385,7 +471,7 @@ class OpportunityEventTests(unittest.TestCase):
|
||||
payload = json.loads(output.read_text(encoding="utf-8"))
|
||||
self.assertEqual(
|
||||
payload["protocol"],
|
||||
"egospeak-opportunity-conditioned-floor-actions-v1",
|
||||
"egospeak-opportunity-conditioned-floor-actions-v2",
|
||||
)
|
||||
self.assertTrue((root / "summary_metrics.csv").is_file())
|
||||
self.assertTrue((root / "summary_events.csv").is_file())
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
"""CPU-only contract tests for MiniCPM-o streaming feature schedules."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
from easycom_head import load_session_data
|
||||
from local_turn_features import reference_alignment
|
||||
from streaming_minicpmo_features_directly import (
|
||||
FEATURE_FORMAT,
|
||||
MiniCPMFeatureExtractor,
|
||||
MiniCPMStreamConfig,
|
||||
_comparison,
|
||||
_time_grid,
|
||||
validate_transformers_version,
|
||||
)
|
||||
|
||||
|
||||
class FakeReader:
|
||||
def __init__(self, duration: float = 3.0) -> None:
|
||||
self.audio = np.arange(
|
||||
int(duration * 16_000), dtype=np.float32
|
||||
) / 16_000.0
|
||||
|
||||
def frame_at_or_before(self, timestamp: float) -> float:
|
||||
return float(timestamp)
|
||||
|
||||
def audio_slice(self, start: float, end: float) -> np.ndarray:
|
||||
first = int(round(start * 16_000))
|
||||
last = int(round(end * 16_000))
|
||||
return self.audio[first:last].copy()
|
||||
|
||||
|
||||
class FakeEngine:
|
||||
hidden_size = 3
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.committed_until = 0.0
|
||||
self.reset_count = 0
|
||||
self.last_restore_audit = None
|
||||
|
||||
def reset(self) -> None:
|
||||
self.committed_until = 0.0
|
||||
self.reset_count += 1
|
||||
|
||||
@staticmethod
|
||||
def _state(frame: float, audio: np.ndarray) -> torch.Tensor:
|
||||
return torch.tensor(
|
||||
[[float(audio[-1]), float(frame), float(len(audio))]],
|
||||
dtype=torch.float32,
|
||||
)
|
||||
|
||||
def commit_state(self, *, frame: float, audio: np.ndarray) -> torch.Tensor:
|
||||
self.committed_until += len(audio) / 16_000.0
|
||||
return self._state(frame, audio)
|
||||
|
||||
def temporary_state(self, *, frame: float, audio: np.ndarray) -> torch.Tensor:
|
||||
before = self.committed_until
|
||||
state = self._state(frame, audio)
|
||||
if self.committed_until != before:
|
||||
raise AssertionError("temporary state modified the fake main cache")
|
||||
self.last_restore_audit = {
|
||||
"kind": "snapshot_restore",
|
||||
"llm_cache_length": 0,
|
||||
"audio_cache_length": 0,
|
||||
"audio_chunk_index": 0,
|
||||
"streaming_position_offset": 0,
|
||||
"window_pruning_suspended": True,
|
||||
"window_policy_restored": True,
|
||||
"llm_checksum_matches": True,
|
||||
"audio_checksum_matches": True,
|
||||
"history_matches": True,
|
||||
"passed": True,
|
||||
}
|
||||
return state
|
||||
|
||||
|
||||
class MiniCPMStreamingTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.config = MiniCPMStreamConfig(
|
||||
boundary_max_abs=0.0,
|
||||
boundary_min_cosine=1.0,
|
||||
replay_max_abs=0.0,
|
||||
replay_min_cosine=1.0,
|
||||
)
|
||||
|
||||
def test_time_grid_is_right_edged_and_rejects_fractional_tail(self) -> None:
|
||||
self.assertEqual(_time_grid(0.0, 1.0, 0.2), [0.2, 0.4, 0.6, 0.8, 1.0])
|
||||
with self.assertRaisesRegex(ValueError, "integer number"):
|
||||
_time_grid(0.0, 1.1, 0.2)
|
||||
|
||||
def test_native_hold_has_no_future_backfill(self) -> None:
|
||||
extractor = MiniCPMFeatureExtractor(FakeReader(), FakeEngine(), self.config)
|
||||
result = extractor.extract_native_hold(
|
||||
start=0.0,
|
||||
end=2.0,
|
||||
show_progress=False,
|
||||
)
|
||||
self.assertEqual(tuple(result.states.shape), (10, 3))
|
||||
self.assertTrue(torch.equal(result.states[:4], torch.zeros(4, 3)))
|
||||
self.assertTrue(result.initialization_mask[:4].all())
|
||||
self.assertFalse(result.initialization_mask[4:].any())
|
||||
self.assertEqual(
|
||||
result.native_update_mask.nonzero().flatten().tolist(),
|
||||
[4, 9],
|
||||
)
|
||||
self.assertTrue(torch.equal(result.states[4], result.states[8]))
|
||||
self.assertFalse(torch.equal(result.states[8], result.states[9]))
|
||||
self.assertEqual(
|
||||
result.state_source_times.tolist(),
|
||||
[0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 2.0],
|
||||
)
|
||||
|
||||
def test_cached_partial_updates_every_row_and_passes_audits(self) -> None:
|
||||
reader = FakeReader()
|
||||
extractor = MiniCPMFeatureExtractor(reader, FakeEngine(), self.config)
|
||||
original_audio = reader.audio.copy()
|
||||
result = extractor.extract_cached_partial(
|
||||
start=0.0,
|
||||
end=2.0,
|
||||
verify_every=0,
|
||||
future_intervention_every=0,
|
||||
allow_unverified=False,
|
||||
show_progress=False,
|
||||
)
|
||||
self.assertTrue(result.partial_tail_admissible)
|
||||
self.assertTrue(np.array_equal(reader.audio, original_audio))
|
||||
self.assertFalse(result.initialization_mask.any())
|
||||
self.assertTrue(torch.equal(result.state_source_times, result.times))
|
||||
self.assertEqual(
|
||||
result.native_update_mask.nonzero().flatten().tolist(),
|
||||
[4, 9],
|
||||
)
|
||||
self.assertEqual(
|
||||
{item["kind"] for item in result.verification},
|
||||
{
|
||||
"native_boundary_equivalence",
|
||||
"clean_stream_replay",
|
||||
"future_audio_intervention",
|
||||
"snapshot_restore",
|
||||
},
|
||||
)
|
||||
self.assertTrue(all(item["passed"] for item in result.verification))
|
||||
self.assertFalse(torch.equal(result.states[0], result.states[1]))
|
||||
self.assertLess(float(result.states[0, 1]), 0.2)
|
||||
|
||||
def test_short_partial_run_is_rejected_without_boundary_audit(self) -> None:
|
||||
extractor = MiniCPMFeatureExtractor(FakeReader(), FakeEngine(), self.config)
|
||||
with self.assertRaisesRegex(RuntimeError, "missing=.*native_boundary"):
|
||||
extractor.extract_cached_partial(
|
||||
start=0.0,
|
||||
end=0.8,
|
||||
verify_every=0,
|
||||
future_intervention_every=0,
|
||||
allow_unverified=False,
|
||||
show_progress=False,
|
||||
)
|
||||
|
||||
def test_comparison_rejects_different_states(self) -> None:
|
||||
item = _comparison(
|
||||
torch.zeros(1, 2),
|
||||
torch.ones(1, 2),
|
||||
kind="test",
|
||||
current_time=0.2,
|
||||
max_abs_tolerance=0.0,
|
||||
min_cosine=1.0,
|
||||
)
|
||||
self.assertFalse(item["passed"])
|
||||
self.assertEqual(item["max_abs"], 1.0)
|
||||
|
||||
def test_transformers_version_is_explicitly_gated(self) -> None:
|
||||
self.assertEqual(validate_transformers_version("4.51.0"), "4.51.0")
|
||||
with self.assertRaisesRegex(RuntimeError, "found 4.52.0"):
|
||||
validate_transformers_version("4.52.0")
|
||||
self.assertEqual(
|
||||
validate_transformers_version("4.52.0", allow_untested=True),
|
||||
"4.52.0",
|
||||
)
|
||||
|
||||
def test_saved_payload_loads_in_existing_head_and_unverified_fails(self) -> None:
|
||||
states = torch.randn(20, 8)
|
||||
payload = {
|
||||
"format": FEATURE_FORMAT,
|
||||
"states": states,
|
||||
"times": (torch.arange(20, dtype=torch.float64) + 1) / 5,
|
||||
"label_indices": torch.arange(20),
|
||||
"mode": "causal-prefix",
|
||||
"extractor_mode": "native-1hz-hold",
|
||||
"config": {"step_seconds": 0.2},
|
||||
}
|
||||
targets = np.tile(
|
||||
np.asarray(
|
||||
[[1, 0, 0], [0, 1, 0], [0, 0, 1], [0, 1, 1]],
|
||||
dtype=np.float32,
|
||||
),
|
||||
(5, 1),
|
||||
)
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
feature_root = root / "features"
|
||||
target_root = root / "targets"
|
||||
feature_root.mkdir()
|
||||
target_root.mkdir()
|
||||
feature_path = feature_root / "Session_4.pt"
|
||||
torch.save(payload, feature_path)
|
||||
np.save(target_root / "Session_4.npy", targets)
|
||||
loaded = load_session_data(feature_root, target_root, 4)
|
||||
self.assertTrue(torch.equal(loaded.states, states))
|
||||
times, indices = reference_alignment(payload)
|
||||
self.assertTrue(torch.equal(times, payload["times"]))
|
||||
self.assertTrue(torch.equal(indices, payload["label_indices"]))
|
||||
|
||||
bad_reference = dict(payload)
|
||||
bad_reference["times"] = payload["times"] + 0.2
|
||||
with self.assertRaisesRegex(ValueError, "Reference times"):
|
||||
reference_alignment(bad_reference)
|
||||
|
||||
payload["mode"] = "minicpmo-unverified-partial-tail"
|
||||
torch.save(payload, feature_path)
|
||||
with self.assertRaisesRegex(ValueError, "expected 'causal-prefix'"):
|
||||
load_session_data(feature_root, target_root, 4)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user