Compare commits

...
5 Commits
14 changed files with 1923 additions and 122 deletions
+25 -13
View File
@@ -54,19 +54,32 @@ result.
- An opportunity-conditioned dense-output evaluator aligns ENTRY positives and
core negatives at non-wearer releases, aligns INT positives and hold-floor
negatives at competing non-wearer onsets, and applies strict competing-risk
censoring. It uses only the original wearer probabilities, excludes wearer
backchannels from headline ENTRY FPR, and records unanchored/ambiguous cases.
censoring. ENTRY positives whose selected release is followed by a competing
other onset or intervening wearer activity are excluded, and ambiguous
multi-competitor INT cases are not assigned an arbitrary onset. It uses only
the original wearer probabilities, excludes wearer backchannels from
headline ENTRY FPR, and records unanchored/ambiguous cases.
### Implemented but not GPU-verified
- MiniCPM-o 4.5 input-only feature extraction has two unmeasured modes: a
native 1 Hz state causally held on the 5 Hz grid, and a 5 Hz completed-cache
plus temporary-partial-tail branch. The latter refuses the normal
head-loading contract unless boundary, replay, snapshot-restoration, and
future-audio checks pass. A real-GPU smoke run is still required; see
[`docs/minicpmo_streaming_features.md`](docs/minicpmo_streaming_features.md).
### Measured screening result
- The released EgoSpeak MiniROADA A+V checkpoint on Sessions 1--3 obtains
`74.124` 0.2-second mAP and `61.155` ten-horizon mean mAP. Raw wearer
onset/offset anticipation is harder (`24.491/38.024` mean AP). On provisional
LLM-assisted matched opportunities, ENTRY AP is `52.487` versus `32.731%`
prevalence, while INT/YIELD AP is `20.135`, indistinguishable from its
`20.438%` prevalence. Figure-7-style continuous wearer curves therefore
support dense forecasting, not semantic floor-control claims. Full curves,
counts, exclusions, and caveats are frozen in
LLM-assisted matched opportunities, the original v1 evaluator reported
ENTRY/INT AP of `52.487/20.135`; those opportunity values are superseded
because v1 could cross a competing ENTRY onset and arbitrarily selected among
multiple INT competitors. Figure-7-style continuous wearer curves support
dense forecasting, not semantic floor-control claims. The unaffected dense
curves and historical v1 opportunity record are in
[`results/easycom_egospeak_three_level_eval_2026-09-16/`](results/easycom_egospeak_three_level_eval_2026-09-16/README.md).
- F1 Frame-set Attention uses fused G+A as one query over the latest half-open
causal person set. On the existing three-seed, four-epoch test screen it
@@ -74,12 +87,11 @@ result.
`64.95099 +/- 0.09451` ten-horizon mean mAP. This is stable, but is not a
matched improvement over the historical no-dev pooled result. Its three-seed
boundary response/yield AP is `28.423 +/- 0.480` / `48.641 +/- 0.391`; on
provisional reviewed opportunities its ENTRY/INT AP is
`41.561 +/- 0.581` / `22.991 +/- 0.202`. These values are respectively
`-4.461/-1.275` points below the historical pooled-V seed-0 result, so the
current F1 screen does not support the hoped-for semantic-selectivity gain.
A same-split three-seed pooled control and correct/wrong visual controls
remain missing. See the immutable
the superseded v1 opportunity evaluator reported ENTRY/INT AP of
`41.561 +/- 0.581` / `22.991 +/- 0.202`; these values require a v2 rerun and
cannot currently support a semantic-selectivity conclusion. A same-split
three-seed pooled control and correct/wrong visual controls remain missing.
See the historical
[`screening record`](results/easycom_omni_floor_screen_2026-09-16/README.md).
### Measured local-fusion screens
+1
View File
@@ -121,6 +121,7 @@ change what the Stage-0 result measures.
| `easycom_manifest.py` | Verified on EasyCom | Discover one-minute clips, preserve session timecodes, and check 5 Hz targets |
| `qwen_omni_features.py` | Implemented, not executed here | Load only the 3B Thinker and extract fused states from one AV window |
| `streaming_turn_model.py` | Verified on EasyCom | Produce one causal Omni state every 200 ms, with causal-prefix and open-stream modes |
| `streaming_minicpmo_features_directly.py` | Implemented; real-GPU smoke pending | Extract input-only MiniCPM-o 4.5 native 1 Hz held states or audited 5 Hz cache-plus-partial-tail states |
| `TurnPredictionHead` | Trained and evaluated | Map `[2048]` to `[10, 3]` logits |
| `train_turn_head.py` | Verified on EasyCom | Train the frozen-Omni anticipation head with the official EasyCom split |
| `infer_turn_head.py` | Verified on EasyCom | Save EgoSpeak-format predictions and reproduce per-horizon AP/mAP |
+23 -14
View File
@@ -10,6 +10,12 @@ head.
A positive must have a reviewer-selected preceding non-wearer release. Its
opportunity starts at that release and resolves at the reviewed wearer entry.
If a new non-wearer onset occurs before or exactly at that entry, the release
has already been resolved by a competing event and the positive is excluded.
The same applies when wearer activity already occurred between the selected
release and the reviewed onset: the selected release is then a stale,
post-outcome anchor. Both cases are counted separately in
`review_diagnostics.skipped`.
Core hard negatives start at the same type of non-wearer release and resolve
when the same speaker resumes (`ENTRY_OTHER_MIDTURN_PAUSE`) or a different
non-wearer takes the next floor (`ENTRY_OTHER_TO_OTHER`). The score is the
@@ -27,10 +33,10 @@ the maximum `1 - p_wearer` over 0.2--0.6 seconds, censored strictly at that
non-wearer's offset. Reviewed `INT_YIELD` items are positive; other
backchannels, failed takeovers, and clear wearer-hold cases are hard negatives.
Current INT reviews do not select one competing onset. The loader therefore
uses the latest supplied other interval that crosses the wearer offset within
the annotation tolerance. It records how many intervals were eligible so
multi-competitor cases can be audited before the protocol is frozen.
Current INT reviews do not select one competing onset. A unique supplied other
interval that crosses the wearer offset within the annotation tolerance is
used. Cases with multiple eligible intervals are excluded and counted rather
than being assigned a deterministic latest-onset anchor.
## Provisional EgoSpeak run
@@ -58,21 +64,24 @@ Before manuscript reporting, freeze the 2.0-second ENTRY cap, 0.6-second YIELD
cap, positive anchor-selection policy, and multiple-competitor handling using
training-side data only.
## Recorded Sessions 1--3 baseline
## Superseded v1 Sessions 1--3 baseline
The first threshold-free run gives ENTRY AP 52.487% (163 positive, 335 hard
negative; 32.731% prevalence) and INT/YIELD AP 20.135% (28 positive, 109 hard
negative; 20.438% prevalence). It excludes 81 ENTRY and 5 INT cases as invalid
or post-outcome. These are provisional LLM-assisted labels, not human-reviewed
confirmatory results. Dense, raw-boundary, and opportunity metrics and their
Figure 7 interpretation are frozen together in the
The first threshold-free run used the superseded v1 loader and reported ENTRY
AP 52.487% (163 positive, 335 hard negative; 32.731% prevalence) and INT/YIELD
AP 20.135% (28 positive, 109 hard negative; 20.438% prevalence). It excluded
81 ENTRY and 5 INT cases as invalid or post-outcome. These opportunity values
must not be reported: v1 allowed a
selected ENTRY release to cross an intervening other onset and selected the
latest onset in ambiguous multi-competitor INT cases. Dense and raw-boundary
metrics are unaffected. Rerun the opportunity evaluation with protocol v2
before updating the manuscript. The historical output is retained in the
[`2026-09-16 result record`](../results/easycom_egospeak_three_level_eval_2026-09-16/README.md).
The expanded Omni screen finds no opportunity-ranking gain from the current
The expanded v1 Omni screen found no opportunity-ranking gain from the current
F1 Frame-set Attention model: its three-seed ENTRY/INT AP is
41.561%/22.991%, below the historical pooled-V seed-0 values
46.022%/24.266%. Because F1 and that pooled run use different train/dev model-
selection protocols, this is a stable negative screen rather than a matched
confirmatory comparison. Dense, boundary, per-seed, and provenance details are
selection protocols and used the superseded opportunity loader, only its dense
and raw-boundary findings remain current. Per-seed and provenance details are
in the
[`Omni floor screening record`](../results/easycom_omni_floor_screen_2026-09-16/README.md).
+204
View File
@@ -0,0 +1,204 @@
# MiniCPM-o 4.5 causal global features
`streaming_minicpmo_features_directly.py` uses MiniCPM-o 4.5 only as a frozen
audio--visual feature extractor. It never calls `streaming_generate`, never
loads TTS, and never lets model-generated speech/listen tokens enter the
conversation history.
## Frozen modes
### `native-1hz-hold`
- The model commits one input-only AV chunk per completed second.
- The newest completed state is forward-held onto the 5 Hz target grid.
- The first four rows (0.2--0.8 s) use an all-zero initialization. The 1.0-s
state is never backfilled into those rows.
- `state_source_times`, `native_update_mask`, and `initialization_mask` make
this alignment auditable.
### `cached-partial-5hz`
- Completed one-second chunks remain in the official MiniCPM streaming cache.
- At a non-boundary 200-ms cutoff, the extractor snapshots the LLM cache,
audio-encoder cache, streaming Mel buffer, chunk index, and RNG state through
MiniCPM's official speculative-state API.
- It appends only the current half-open AV tail, reads the final normalized
Qwen3 hidden state, and restores the snapshot. Temporary closure never enters
the main cache.
- Native commits retain MiniCPM's normal streaming-window policy. Destructive
round pruning is suspended only inside the temporary transaction because the
upstream snapshot can truncate appended KV tokens but cannot recreate a
previously pruned round.
- At a one-second boundary, a complete native chunk is committed.
The 5 Hz branch is saved as the normal `mode=causal-prefix` contract only after
all mandatory checks pass:
1. a temporary full chunk matches the native committed state at a one-second
boundary;
2. a clean stream replay matches the cached result;
3. changing audio strictly after the cutoff leaves the current state exactly
unchanged;
4. snapshot restoration leaves LLM/audio cache lengths and checksums, chunk
history, position offset, and the audio chunk index unchanged.
With `--allow-unverified-partial-tail`, a failed or too-short audit may be
saved only as `mode=minicpmo-unverified-partial-tail`; the default head loader
rejects that artifact.
## Environment
The inspected upstream revision requires Python 3.10 and
`transformers==4.51.0`. Use a dedicated environment rather than changing the
working Qwen environment.
```bash
python -m pip install \
"transformers==4.51.0" \
accelerate \
"torch>=2.3.0,<=2.8.0" \
"torchaudio<=2.8.0" \
"minicpmo-utils>=1.0.5" \
"qwen-omni-utils[decord]"
```
The commands below use the upstream revision inspected during implementation,
`1f761131fa83f5ed3cd6f2f22b225c4501d154fa`. If the model is already stored
locally, replace `--model-id` with that snapshot directory and record its
immutable revision separately.
## Required smoke sequence
First validate the source timeline and decoder without loading MiniCPM:
```bash
python streaming_minicpmo_features_directly.py \
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
--end 3 \
--mode native-1hz-hold \
--decoder decord \
--media-only
```
Then extract three seconds with the native held-state baseline:
```bash
CUDA_VISIBLE_DEVICES=0 python streaming_minicpmo_features_directly.py \
--model-id openbmb/MiniCPM-o-4_5 \
--revision 1f761131fa83f5ed3cd6f2f22b225c4501d154fa \
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
--end 3 \
--mode native-1hz-hold \
--dtype bfloat16 \
--attention sdpa \
--device cuda:0 \
--decoder decord \
--decoder-threads 1 \
--max-open-readers 1 \
--output /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/smoke/Session_4_native_3s.pt
```
Run the verified cached partial-tail smoke test separately:
```bash
CUDA_VISIBLE_DEVICES=0 python streaming_minicpmo_features_directly.py \
--model-id openbmb/MiniCPM-o-4_5 \
--revision 1f761131fa83f5ed3cd6f2f22b225c4501d154fa \
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
--end 3 \
--mode cached-partial-5hz \
--verify-every 1 \
--future-intervention-every 1 \
--dtype bfloat16 \
--attention sdpa \
--device cuda:0 \
--decoder decord \
--decoder-threads 1 \
--max-open-readers 1 \
--output /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/smoke/Session_4_partial_3s.pt
```
Do not add `--allow-unverified-partial-tail` to a paper run. A verification
failure is a protocol failure to diagnose, not a tolerance to bypass.
Inspect both artifacts before a full session:
```bash
python - <<'PY'
from pathlib import Path
import torch
root = Path("/data1/yfzx/wangx/Datasets/easycom/minicpmo_features/smoke")
for name in ("Session_4_native_3s.pt", "Session_4_partial_3s.pt"):
payload = torch.load(root / name, map_location="cpu", weights_only=False)
print(name)
print(" mode:", payload["mode"], payload["extractor_mode"])
print(" states:", tuple(payload["states"].shape))
print(" times:", payload["times"][:6].tolist())
print(" source times:", payload["state_source_times"][:6].tolist())
print(" verified:", payload["partial_tail_admissible"])
print(" checks:", payload["verification"])
PY
```
Expected alignment:
```text
native times: 0.2 0.4 0.6 0.8 1.0 1.2 ...
native source time: 0.0 0.0 0.0 0.0 1.0 1.0 ...
partial source time: 0.2 0.4 0.6 0.8 1.0 1.2 ...
```
## Full Session 4
Native 1 Hz held features:
```bash
mkdir -p /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/native_1hz_hold
CUDA_VISIBLE_DEVICES=0 python streaming_minicpmo_features_directly.py \
--model-id openbmb/MiniCPM-o-4_5 \
--revision 1f761131fa83f5ed3cd6f2f22b225c4501d154fa \
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
--target /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame/Session_4.npy \
--mode native-1hz-hold \
--dtype bfloat16 \
--attention sdpa \
--device cuda:0 \
--decoder decord \
--decoder-threads 1 \
--max-open-readers 1 \
--frame-cache-size 32 \
--output /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/native_1hz_hold/Session_4.pt
```
Verified 5 Hz cached partial-tail features:
```bash
mkdir -p /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/cached_partial_5hz
CUDA_VISIBLE_DEVICES=0 python streaming_minicpmo_features_directly.py \
--model-id openbmb/MiniCPM-o-4_5 \
--revision 1f761131fa83f5ed3cd6f2f22b225c4501d154fa \
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
--target /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame/Session_4.npy \
--mode cached-partial-5hz \
--dtype bfloat16 \
--attention sdpa \
--device cuda:0 \
--decoder decord \
--decoder-threads 1 \
--max-open-readers 1 \
--frame-cache-size 32 \
--output /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/cached_partial_5hz/Session_4.pt
```
The full run still executes the first clean-replay and future-intervention
checks plus every one-second boundary check. Add `--verify-every 250` and
`--future-intervention-every 250` for periodic 50-second audits; smaller values
make long sessions increasingly expensive because a clean audit replays the
prefix from the session start.
After extraction, either directory can be passed directly as
`train_turn_head.py --feature-root ...` because verified artifacts retain the
existing dense global-feature contract.
@@ -168,12 +168,13 @@ forecasting 的定性解释,不能代替 boundary 或 semantic opportunity 评
曲线和限制见
[`three-level baseline record`](../results/easycom_egospeak_three_level_eval_2026-09-16/README.md)。
当前 F1 current-frame person-set screen 的三种子结果同样没有支持 semantic floor
selectivity:ENTRY/INT opportunity AP 为 `41.561 +/- 0.581` / `22.991 +/- 0.202`,
低于历史 pooled-V seed-0 的 `46.022/24.266`;dense mean mAP 也为
`64.951 +/- 0.095`,没有超过历史 pooled-V 的 `65.400`。由于两者 train/dev
选择协议不同,这只能判为稳定的负向 screening,正式归因前必须补同协议三种子 pooled
control。详见 [`Omni floor screening record`](../results/easycom_omni_floor_screen_2026-09-16/README.md)。
当前 F1 current-frame person-set screen 的 dense mean mAP 为
`64.951 +/- 0.095`,没有超过历史 pooled-V 的 `65.400`。原 v1 evaluator 给出的
ENTRY/INT opportunity AP(`41.561 +/- 0.581` / `22.991 +/- 0.202`)已失效:v1
可能跨过 competing ENTRY onset,且会在多个 INT competitor 中任意选择 latest onset。
必须用 v2 重跑后才能判断 semantic floor selectivity;正式归因还必须补同协议三种子
pooled control。历史数值与勘误见
[`Omni floor screening record`](../results/easycom_omni_floor_screen_2026-09-16/README.md)。
### P3:第一优先方法——匿名逐人时序旁路
+136 -25
View File
@@ -8,14 +8,18 @@ event head.
ENTRY opportunities are anchored at a reviewed preceding non-wearer release.
Positive opportunities resolve at the wearer entry; core hard negatives
resolve when the same or a different non-wearer speaks next. The score is the
maximum wearer probability over available 0.2--2.0 s forecasts up to that
competing resolution.
resolve when the same or a different non-wearer speaks next. A positive is
excluded if another non-wearer onset competes before (or exactly at) its wearer
entry, or if wearer activity already occurred after the selected release. The
score is the maximum wearer probability over available 0.2--2.0 s forecasts up
to the uncontested resolution.
YIELD opportunities are anchored at a non-wearer onset during wearer speech.
The score is the maximum future wearer-inactivity probability over 0.2--0.6 s,
censored at that non-wearer's offset. Positive and hard-negative examples
therefore begin at the same kind of observed conversational opportunity.
censored at that non-wearer's offset. Reviews with multiple eligible
competing onsets are excluded until the annotation selects one. Positive and
hard-negative examples therefore begin at the same kind of unambiguous
observed conversational opportunity.
Prediction arrays use the repository contract
``[target_time, class, future_horizon]``. All scores are read diagonally from
@@ -178,17 +182,83 @@ def _candidate_region(
return start, end
def _int_competing_interval(
def _entry_selected_option(
candidate: dict[str, Any],
review: dict[str, Any],
*,
selected_release_seconds: float,
uid: str,
) -> dict[str, Any] | None:
evidence = candidate.get("temporal_evidence")
evidence = evidence if isinstance(evidence, dict) else {}
options = evidence.get("opportunity_options")
if not isinstance(options, list):
return None
selected_uid = review.get("selected_opportunity_uid")
matches: list[dict[str, Any]] = []
for index, option in enumerate(options):
if not isinstance(option, dict):
continue
option_seconds = _optional_seconds(
option.get("anchor_seconds"),
field=f"opportunity_options[{index}].anchor_seconds",
uid=uid,
)
if selected_uid is not None and option.get("uid") == selected_uid:
matches.append(option)
elif selected_uid is None and option_seconds is not None and math.isclose(
option_seconds, selected_release_seconds, abs_tol=1e-7
):
matches.append(option)
if len(matches) > 1:
raise ValueError(f"{uid}: selected ENTRY opportunity is not unique")
return matches[0] if matches else None
def _entry_competing_onsets(
candidate: dict[str, Any],
*,
selected_release_seconds: float,
behavior_anchor_seconds: float,
uid: str,
) -> list[float]:
"""Return other onsets that resolve a release before the wearer entry."""
evidence = candidate.get("temporal_evidence")
evidence = evidence if isinstance(evidence, dict) else {}
raw_intervals = evidence.get("gap_other_intervals")
if not isinstance(raw_intervals, list):
raw_intervals = []
onsets: set[float] = set()
for index, interval in enumerate(raw_intervals):
if not isinstance(interval, dict):
continue
onset = _optional_seconds(
interval.get("onset_seconds"),
field=f"gap_other_intervals[{index}].onset_seconds",
uid=uid,
)
if onset is None:
continue
if (
onset >= selected_release_seconds - 1e-7
and onset <= behavior_anchor_seconds + 1e-7
):
onsets.add(onset)
return sorted(onsets)
def _int_competing_intervals(
candidate: dict[str, Any],
*,
behavior_anchor_seconds: float,
uid: str,
) -> tuple[float, float, int] | None:
) -> list[tuple[float, float]]:
evidence = candidate.get("temporal_evidence")
evidence = evidence if isinstance(evidence, dict) else {}
raw_intervals = evidence.get("nearby_other_intervals")
if not isinstance(raw_intervals, list):
return None
return []
eligible: list[tuple[float, float]] = []
for index, interval in enumerate(raw_intervals):
if not isinstance(interval, dict):
@@ -209,13 +279,7 @@ def _int_competing_interval(
continue
if onset < behavior_anchor_seconds and offset > onset:
eligible.append((onset, offset))
if not eligible:
return None
# A semantic INT review does not currently select one competing onset.
# The latest interval crossing the wearer offset is the deterministic,
# most proximal causal candidate. Multiplicity is surfaced for audit.
onset, offset = max(eligible, key=lambda item: (item[0], item[1]))
return onset, offset, len(eligible)
return sorted(set(eligible))
def load_opportunity_events(
@@ -278,6 +342,38 @@ def load_opportunity_events(
if selected_release is None:
skipped["entry_positive_without_selected_release"] += 1
continue
selected_option = _entry_selected_option(
candidate,
review,
selected_release_seconds=selected_release,
uid=uid,
)
if (
selected_option is not None
and selected_option.get(
"wearer_active_between_release_and_onset"
) is True
):
skipped["entry_positive_post_outcome_wearer_activity"] += 1
continue
competing_onsets = _entry_competing_onsets(
candidate,
selected_release_seconds=selected_release,
behavior_anchor_seconds=behavior,
uid=uid,
)
option_reports_competitor = (
selected_option is not None
and selected_option.get(
"other_onset_between_release_and_onset"
) is True
)
if option_reports_competitor and not competing_onsets:
skipped["entry_positive_competing_onset_unresolved"] += 1
continue
if competing_onsets:
skipped["entry_positive_competed_before_behavior"] += 1
continue
anchor = selected_release
resolution = behavior
censor = behavior
@@ -298,17 +394,21 @@ def load_opportunity_events(
competing_intervals = 0
else:
if positive:
interval = _int_competing_interval(
intervals = _int_competing_intervals(
candidate,
behavior_anchor_seconds=behavior,
uid=uid,
)
if interval is None:
if not intervals:
skipped["int_positive_without_crossing_other"] += 1
continue
anchor, censor, competing_intervals = interval
if len(intervals) > 1:
skipped["int_positive_ambiguous_competing_others"] += 1
continue
anchor, censor = intervals[0]
competing_intervals = 1
resolution = behavior
basis = "latest_other_onset_crossing_wearer_offset"
basis = "unique_other_onset_crossing_wearer_offset"
elif subtype_value in INT_HARD_NEGATIVES:
if source in INT_MINED_SOURCES:
anchor, censor = _candidate_region(candidate, uid=uid)
@@ -316,17 +416,21 @@ def load_opportunity_events(
competing_intervals = 1
basis = "mined_competing_other_onset"
else:
interval = _int_competing_interval(
intervals = _int_competing_intervals(
candidate,
behavior_anchor_seconds=behavior,
uid=uid,
)
if interval is None:
if not intervals:
skipped["int_hn_without_competing_other"] += 1
continue
anchor, censor, competing_intervals = interval
if len(intervals) > 1:
skipped["int_hn_ambiguous_competing_others"] += 1
continue
anchor, censor = intervals[0]
competing_intervals = 1
resolution = behavior
basis = "latest_other_onset_crossing_non_yield_offset"
basis = "unique_other_onset_crossing_non_yield_offset"
else:
skipped["int_unsupported_hard_negative"] += 1
continue
@@ -731,8 +835,8 @@ def main() -> None:
else calibration_diagnostics["num_provisional_llm"]
)
payload = {
"schema_version": 1,
"protocol": "egospeak-opportunity-conditioned-floor-actions-v1",
"schema_version": 2,
"protocol": "egospeak-opportunity-conditioned-floor-actions-v2",
"semantic_scope": (
"LLM-assisted opportunity labels with dense wearer-activity scores; "
"this is provisional and no human review is implied."
@@ -757,6 +861,13 @@ def main() -> None:
"yield_max_seconds": yield_max,
"entry_headline_hard_negatives": sorted(ENTRY_CORE_HARD_NEGATIVES),
"entry_backchannel_policy": "separate diagnostic; excluded from headline",
"entry_competing_risk_policy": (
"exclude selected-release positives with an other onset at or before "
"the wearer entry, or with intervening wearer activity"
),
"int_multiple_competitor_policy": (
"exclude until one competing onset is selected by review"
),
"matched_recalls": recalls,
"review_diagnostics": review_diagnostics,
"calibration_review_diagnostics": calibration_diagnostics,
+86 -41
View File
@@ -339,9 +339,12 @@ non-wearer release. It resolves positively when the wearer enters, or
negatively when the same non-wearer resumes or a different non-wearer takes the
next floor. From the first causal 5-Hz state that observes the release, the
entry score is the maximum projected wearer probability before that competing
resolution, capped at 2.0~s. Positive entries without a reviewer-selected
release are reported as unanchored and excluded from this matched-opportunity
analysis; they remain available to the separate raw-onset analysis.
resolution, capped at 2.0~s. A selected-release positive is excluded if another
non-wearer onset occurs before or exactly at the wearer entry, or if wearer
activity already occurred after the selected release. Positive entries without
a reviewer-selected release are reported as unanchored and excluded from this
matched-opportunity analysis; they remain available to the separate raw-onset
analysis.
An INT opportunity begins when a non-wearer starts while the wearer holds the
floor. It resolves positively when the reviewed behavior is an INT-yield, and
@@ -352,8 +355,9 @@ strictly at the competing non-wearer offset. All prediction values are read
diagonally from one causal source row of the target-aligned forecast tensor.
No input after the observed action outcome is used. For current INT-positive
annotations, which do not select a competing onset explicitly, the
implementation uses the latest supplied non-wearer interval crossing the
wearer offset and surfaces cases with multiple eligible intervals for audit.
implementation evaluates only a unique supplied non-wearer interval crossing
the wearer offset; cases with multiple eligible intervals are excluded and
counted for audit rather than assigned an arbitrary onset.
The headline entry hard negatives are same-speaker mid-turn pauses and other-
to-other transfers. Wearer backchannels are not pooled into headline false-
@@ -394,46 +398,60 @@ latency claim.
\subsection{Main Comparison}
Table~\ref{tab:main} compares the proposed causal local audiovisual adapter
with an EgoSpeak reproduction, the frozen Qwen2.5-Omni-3B global state $G$,
matched PEFT, $G$ plus recent local audio, and a parameter- and history-matched
temporal local branch. Qwen2.5-Omni-7B is included only as a capacity-scaling
control. All methods use the same split, causal inputs, targets, and evaluation
protocol. Event thresholds are selected on training-side calibration data and
frozen before test evaluation. \todo{Freeze all baseline implementations and
verify identical targets, causal validity, and evaluation.}
Table~\ref{tab:main} places the released EgoSpeak checkpoint beside the
currently available Omni screening runs and reserves a row for the final causal
local audiovisual model. Historical unmatched runs are identified explicitly
and cannot support confirmatory attribution. The final comparison will use the
same split, causal inputs, targets, training/selection protocol, and three
random seeds; event thresholds will be selected on training-side calibration
data and frozen before test evaluation. \todo{Freeze all baseline
implementations and verify identical targets, causal validity, and evaluation.}
\begin{table}[t]
\caption{Confirmatory EasyCom results. Entry and INT false-positive rates use
the opportunity-conditioned activity score at matched recall $R=0.8$; E/I AP
is the corresponding opportunity AP. $\dagger$ denotes provisional
LLM-assisted labels pending human review.}
\begin{table*}[t]
\caption{EasyCom Sessions~1--3 dense future-activity and raw-boundary
screening. All values are percentages. ``Avg.'' averages dense mAP over the ten
horizons; On/Off are mean raw wearer onset/offset AP and are timing diagnostics,
not semantic ENTRY/INT scores. $\dagger$ denotes historical seed-0 runs with an
unmatched training/selection protocol; $\ddagger$ denotes a three-seed screen.
Missing horizon curves and the final matched comparison will be filled after
the matched dense/boundary reruns.}
\label{tab:main}
\centering
\small
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lccccc}
\scriptsize
\setlength{\tabcolsep}{2.6pt}
\resizebox{\textwidth}{!}{%
\begin{tabular}{@{}lccccccccccccc@{}}
\hline
Method & 0.2-s mAP $\uparrow$ & Mean mAP $\uparrow$
& Entry FPR@$R=.8$ $\downarrow$ & INT FPR@$R=.8$ $\downarrow$ & E/I AP $\uparrow$\\
& \multicolumn{10}{c}{Dense mAP at prediction horizon (s) $\uparrow$}
& & \multicolumn{2}{c}{Raw boundary AP $\uparrow$}\\
\cline{2-11}\cline{13-14}
Method & .2 & .4 & .6 & .8 & 1.0 & 1.2 & 1.4 & 1.6 & 1.8 & 2.0
& Avg. & On & Off\\
\hline
EgoSpeak MiniROADA A+V & 74.12 & 61.16 & \todo{--} & \todo{--} & $52.49/20.13^\dagger$\\
Omni-3B frozen $G$ & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
Omni-3B matched PEFT & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
$G+$local audio & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
$G+$matched temporal local & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
$G+$causal local AV (ours) & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
Omni-7B scaling control & \todo{--} & \todo{--} & \todo{--} & \todo{--} & \todo{--/--}\\
EgoSpeak MiniROADA A+V
& 74.20 & 68.35 & 64.31 & 61.13 & 59.77 & 58.81 & 57.50 & 57.09 & 55.95 & 54.82
& 61.19 & 24.49 & 38.02\\
Omni $G{+}A^{\dagger}$
& 81.72 & -- & -- & -- & -- & -- & -- & -- & -- & --
& 65.11 & 25.96 & 45.25\\
Omni $G{+}A{+}$pooled-$V^{\dagger}$
& 82.03 & -- & -- & -- & -- & -- & -- & -- & -- & --
& 65.40 & 26.02 & 45.37\\
Current-frame F1$^{\ddagger}$
& $81.78{\pm}0.16$ & -- & -- & -- & -- & -- & -- & -- & -- & --
& $64.95{\pm}0.09$ & -- & --\\
$G+$causal local AV (ours)
& -- & -- & -- & -- & -- & -- & -- & -- & -- & --
& -- & -- & --\\
\hline
\end{tabular}}
\end{table}
\end{table*}
\textbf{Dense forecasting.}
The EgoSpeak baseline reaches 74.12\% 0.2-s mAP and 61.16\% mean mAP; its raw
wearer-onset/offset mean APs are 24.49/38.02\%. On provisional reviewed
opportunities, ENTRY AP is 52.49\% versus 32.73\% prevalence, whereas INT AP is
20.13\% versus 20.44\%; continuous wearer-score plots therefore do not alone
establish semantic floor-yield selectivity.
The EgoSpeak baseline reaches 74.20\% 0.2-s mAP and 61.19\% mean mAP; its raw
wearer-onset/offset mean APs are 24.49/38.02\%. Opportunity-conditioned values
remain pending a protocol-v2 rerun and human review; continuous wearer-score
plots do not alone establish semantic floor-yield selectivity.
The proposed local AV adapter improves 0.2-s mAP from
\todo{$X$ to $Y$} over the strongest matched baseline and improves
mean mAP by \todo{$Z$ points}.
@@ -460,6 +478,34 @@ beyond additional temporal capacity}.
\todo{If this difference is not statistically supported, state instead that
the local temporal branch accounts for most of the gain.}
\subsection{Qualitative Floor-Event Analysis}
We will compare EgoSpeak with the final matched model on human-confirmed
examples selected by pre-declared event subtype rather than post-hoc score
extremes. Each case will align participant-resolved activity, minimal transcript
context, the opportunity anchor and competing-risk censor, and the two models'
causally emitted wearer forecasts. The comparison will include a genuine wearer
entry, an other-to-other false-entry opportunity, and an INT-yield paired with a
non-takeover overlap; a shared failure case will be retained when space permits.
\begin{figure*}[t]
\centering
\fbox{\parbox[c][0.115\textheight][c]{0.965\textwidth}{
\centering
\textbf{Qualitative comparison placeholder}\\[0.7em]
\begin{tabular}{ccc}
Case A: wearer entry & Case B: other$\rightarrow$other HN
& Case C: INT-yield / failed takeover\\
participant timeline + transcript & opportunity anchor + censor
& EgoSpeak vs. proposed forecast\\
\end{tabular}
}}
\caption{Planned qualitative floor-event comparison. Final panels will use
human-confirmed protocol-v2 cases and show the same causal source time for both
models; the figure is a layout placeholder and contains no result claim.}
\label{fig:qualitative_floor_events}
\end{figure*}
\subsection{Information Opportunity}
Before designing the deployable adapter, we measured whether local floor
@@ -570,7 +616,7 @@ future-media intervention test. Report within-backbone 5-Hz versus 1-Hz
deltas plus cache/no-cache throughput and memory; do not interpret unmatched
Qwen--MiniCPM raw scores as a pure backbone effect.}
\subsection{Limitations and Qualitative Analysis}
\subsection{Limitations}
The results support \todo{precise supported claim}, but several limitations
remain.
@@ -580,10 +626,9 @@ tracking assumption}.
Third, \todo{failure condition: occlusion, overlapping speakers, rapid floor
exchange, etc.} remains challenging.
\todo{Add one compact, pre-selected qualitative timeline only if space permits:
one multi-other false-entry improvement paired with one preserved real wearer
onset or one failure. Examples illustrate aggregate rates and never establish
the claim by themselves.}
\todo{Replace Fig.~\ref{fig:qualitative_floor_events} with compact, pre-selected
human-confirmed timelines. Examples illustrate aggregate rates and never
establish the claim by themselves.}
\section{Conclusion}
\label{sec:conclusion}
@@ -1,5 +1,11 @@
# EasyCom EgoSpeak three-level evaluation
> **Protocol erratum.** The dense and raw-boundary results in this record remain
> valid. The opportunity-conditioned ENTRY/INT values were produced by the
> superseded v1 loader, which could cross an intervening ENTRY competitor and
> selected the latest onset in ambiguous multi-competitor INT cases. Do not cite
> those opportunity values; rerun protocol v2 and create a new result record.
This record freezes the first Sessions 1--3 evaluation of the released
EgoSpeak MiniROADA audio-visual checkpoint under three deliberately separate
questions. The values below were produced on the remote experiment host; the
@@ -15,7 +21,7 @@ prediction and per-event artifacts remain on that host and are not committed.
- Raw boundary anticipation: threshold-free AP for wearer `OFF->ON` and
`ON->OFF` changes. These are timing-derived diagnostics, not semantic EOT or
INT claims.
- Opportunity-conditioned behavior: one causally censored activity-derived
- Opportunity-conditioned behavior: historical v1 activity-derived
score per reviewed ENTRY or INT/YIELD opportunity. Labels are provisional
two-pass LLM-assisted reviews and have not been human reviewed.
- No test-set threshold is fitted for any AP reported here. Session 4 is needed
@@ -34,15 +40,16 @@ mean AP, respectively. These lower values do not contradict the dense result:
dense AP rewards correct future state over sustained regions, whereas a raw
boundary event is counted once and has many harder non-boundary comparisons.
On the provisional opportunity labels, ENTRY AP is **52.487%** for 163
The superseded v1 run reported ENTRY AP of **52.487%** for 163
positives and 335 hard negatives, versus 32.731% positive prevalence
(+19.756 points; 1.60x prevalence). INT/YIELD AP is **20.135%** for 28 positives
and 109 hard negatives, essentially equal to its 20.438% prevalence. This is
consistent with the output contract: wearer probability can rank many entry
opportunities, but `1-p(wearer)` does not encode whether an interruption should
cause a yield. The ENTRY and INT loaders additionally exclude 81 and 5
invalid/post-outcome cases; these exclusions require an audit before any final
paper claim.
and 109 hard negatives, essentially equal to its 20.438% prevalence. These
values are retained only for provenance and must be replaced by a v2 rerun. The
original pattern was consistent with the output contract: wearer probability
can rank many entry opportunities, but `1-p(wearer)` does not encode whether an
interruption should cause a yield. The ENTRY and INT loaders additionally
exclude 81 and 5 invalid/post-outcome cases; these exclusions require an audit
before any final paper claim.
## Relation to EgoSpeak Figure 7
@@ -73,4 +80,3 @@ Machine-readable curves and counts are in [`summary.json`](summary.json) and
the compact comparison is in [`metrics.csv`](metrics.csv). The opportunity
protocol and reproduction command are documented in
[`docs/easycom_opportunity_event_evaluation.md`](../../docs/easycom_opportunity_event_evaluation.md).
@@ -1,6 +1,8 @@
{
"recorded_at": "2026-09-16",
"source": "user-provided remote experiment output",
"opportunity_protocol_status": "superseded_v1_rerun_required",
"opportunity_protocol_erratum": "Dense and raw-boundary metrics remain valid; opportunity metrics must be rerun with egospeak-opportunity-conditioned-floor-actions-v2.",
"dataset": "EasyCom",
"sessions": ["Session_1", "Session_2", "Session_3"],
"model": "EgoSpeak MiniROADA audio-visual released checkpoint",
@@ -57,4 +59,3 @@
"threshold_free": true,
"session4_required_for_reported_metrics": false
}
@@ -1,5 +1,12 @@
# EasyCom Omni floor-event screening
> **Protocol erratum.** Dense and raw-boundary metrics remain valid. All
> opportunity-conditioned ENTRY/INT values below used the superseded v1 loader,
> which could cross an intervening ENTRY competitor and selected the latest
> onset in ambiguous multi-competitor INT cases. Retain them for provenance
> only; do not use them for model selection or manuscript claims before a v2
> rerun.
This record freezes the first comparison of EgoSpeak, Omni `G+A`, Omni
`G+A+pooled-V`, and the three-seed current-frame F1 Frame-set Attention screen
on EasyCom Sessions 1--3. Values were produced on the remote experiment host
@@ -11,9 +18,8 @@ CSVs remain on that host and are not committed here.
1. Dense future activity uses causal-valid framewise AP/mAP over ten horizons.
2. Raw wearer endpoint and aggregate-other-boundary AP are threshold-free
timing diagnostics; they are not semantic EOT/INT labels.
3. Opportunity-conditioned ENTRY and INT/YIELD AP assigns one causally
censored activity-derived score to each provisional LLM-assisted reviewed
opportunity.
3. Opportunity-conditioned ENTRY and INT/YIELD AP is a historical v1 screen
that requires rerunning with the corrected v2 competing-risk protocol.
All opportunity runs use identical denominators: 163 ENTRY positives and 335
hard negatives, with 81 invalid/post-outcome cases; and 28 INT positives and
@@ -39,7 +45,7 @@ from EgoSpeak to G+A to G+A+pooled-V as follows: onset AP
offset/yield-timing gain comes from G+A; pooled visual input adds almost
nothing to raw transition timing.
## Interpretation
## Historical v1 interpretation
- G+A strongly improves dense forecasting and generic wearer-offset/yield
timing over EgoSpeak, but not reviewed semantic opportunities. In particular,
@@ -77,4 +83,3 @@ nothing to raw transition timing.
Exact aggregate values are stored in [`summary.json`](summary.json) and
[`metrics.csv`](metrics.csv). The earlier EgoSpeak-only record is
[`easycom_egospeak_three_level_eval_2026-09-16`](../easycom_egospeak_three_level_eval_2026-09-16/README.md).
@@ -1,6 +1,8 @@
{
"recorded_at": "2026-09-16",
"source": "user-provided remote experiment output",
"opportunity_protocol_status": "superseded_v1_rerun_required",
"opportunity_protocol_erratum": "Dense and raw-boundary metrics remain valid; opportunity metrics must be rerun with egospeak-opportunity-conditioned-floor-actions-v2.",
"dataset": "EasyCom",
"test_sessions": ["Session_1", "Session_2", "Session_3"],
"scale": "percent",
@@ -82,4 +84,3 @@
"threshold_free_metrics_only": true,
"matched_pooled_control_available": false
}
File diff suppressed because it is too large Load Diff
+93 -7
View File
@@ -30,7 +30,10 @@ def reviewed_row(
region_start: float,
region_end: float,
selected_release: float | None = None,
selected_release_uid: str | None = None,
nearby: list[dict[str, object]] | None = None,
opportunity_options: list[dict[str, object]] | None = None,
gap_other_intervals: list[dict[str, object]] | None = None,
candidate_source: str | None = None,
) -> dict[str, object]:
event_type = None
@@ -45,6 +48,7 @@ def reviewed_row(
"human_verdict": verdict,
"human_event_type": event_type,
"human_subtype": subtype,
"selected_opportunity_uid": selected_release_uid,
"selected_opportunity_seconds": selected_release,
"boundary_correction_seconds": 0,
},
@@ -59,6 +63,8 @@ def reviewed_row(
"behavior_anchor_seconds": anchor,
"wearer_offset_seconds": anchor,
"nearby_other_intervals": nearby or [],
"opportunity_options": opportunity_options or [],
"gap_other_intervals": gap_other_intervals or [],
},
},
}
@@ -119,7 +125,7 @@ class OpportunityEventTests(unittest.TestCase):
diagnostics["skipped"]["entry_backchannel_separate_diagnostic"], 1
)
def test_loader_selects_latest_crossing_other_for_int(self) -> None:
def test_loader_excludes_ambiguous_crossing_others_for_int(self) -> None:
nearby = [
{
"onset_seconds": 1.0,
@@ -145,11 +151,91 @@ class OpportunityEventTests(unittest.TestCase):
)
])
self.addCleanup(temporary.cleanup)
events, _ = load_opportunity_events([path], ["Session_4"])
self.assertEqual(len(events), 1)
self.assertEqual(events[0].opportunity_anchor_seconds, 2.0)
self.assertEqual(events[0].censor_seconds, 3.8)
self.assertEqual(events[0].competing_intervals, 2)
events, diagnostics = load_opportunity_events([path], ["Session_4"])
self.assertEqual(events, [])
self.assertEqual(
diagnostics["skipped"]["int_positive_ambiguous_competing_others"],
1,
)
def test_loader_excludes_entry_positive_at_competing_other_onset(self) -> None:
rows = [
reviewed_row(
"entry-competed",
family="entry",
verdict="POSITIVE",
subtype=None,
anchor=1.6,
region_start=0.0,
region_end=3.0,
selected_release=1.0,
selected_release_uid="release",
opportunity_options=[
{
"uid": "release",
"anchor_seconds": 1.0,
"wearer_active_between_release_and_onset": False,
"other_onset_between_release_and_onset": True,
}
],
gap_other_intervals=[
{"uid": "competitor", "onset_seconds": 1.4}
],
),
reviewed_row(
"entry-simultaneous",
family="entry",
verdict="POSITIVE",
subtype=None,
anchor=4.0,
region_start=3.0,
region_end=5.0,
selected_release=3.2,
gap_other_intervals=[
{"uid": "competitor", "onset_seconds": 4.0}
],
),
]
temporary, path = self.write_reviews(rows)
self.addCleanup(temporary.cleanup)
events, diagnostics = load_opportunity_events([path], ["Session_4"])
self.assertEqual(events, [])
self.assertEqual(
diagnostics["skipped"]["entry_positive_competed_before_behavior"],
2,
)
def test_loader_excludes_entry_positive_after_prior_wearer_activity(self) -> None:
temporary, path = self.write_reviews([
reviewed_row(
"entry-post-outcome",
family="entry",
verdict="POSITIVE",
subtype=None,
anchor=2.0,
region_start=0.0,
region_end=3.0,
selected_release=1.0,
selected_release_uid="release",
opportunity_options=[
{
"uid": "release",
"anchor_seconds": 1.0,
"wearer_active_between_release_and_onset": True,
"other_onset_between_release_and_onset": False,
}
],
)
])
self.addCleanup(temporary.cleanup)
events, diagnostics = load_opportunity_events([path], ["Session_4"])
self.assertEqual(events, [])
self.assertEqual(
diagnostics["skipped"][
"entry_positive_post_outcome_wearer_activity"
],
1,
)
def test_entry_hn_requires_matching_independently_mined_source(self) -> None:
rows = [
@@ -385,7 +471,7 @@ class OpportunityEventTests(unittest.TestCase):
payload = json.loads(output.read_text(encoding="utf-8"))
self.assertEqual(
payload["protocol"],
"egospeak-opportunity-conditioned-floor-actions-v1",
"egospeak-opportunity-conditioned-floor-actions-v2",
)
self.assertTrue((root / "summary_metrics.csv").is_file())
self.assertTrue((root / "summary_events.csv").is_file())
@@ -0,0 +1,230 @@
"""CPU-only contract tests for MiniCPM-o streaming feature schedules."""
from __future__ import annotations
import tempfile
import unittest
from pathlib import Path
import numpy as np
import torch
from easycom_head import load_session_data
from local_turn_features import reference_alignment
from streaming_minicpmo_features_directly import (
FEATURE_FORMAT,
MiniCPMFeatureExtractor,
MiniCPMStreamConfig,
_comparison,
_time_grid,
validate_transformers_version,
)
class FakeReader:
def __init__(self, duration: float = 3.0) -> None:
self.audio = np.arange(
int(duration * 16_000), dtype=np.float32
) / 16_000.0
def frame_at_or_before(self, timestamp: float) -> float:
return float(timestamp)
def audio_slice(self, start: float, end: float) -> np.ndarray:
first = int(round(start * 16_000))
last = int(round(end * 16_000))
return self.audio[first:last].copy()
class FakeEngine:
hidden_size = 3
def __init__(self) -> None:
self.committed_until = 0.0
self.reset_count = 0
self.last_restore_audit = None
def reset(self) -> None:
self.committed_until = 0.0
self.reset_count += 1
@staticmethod
def _state(frame: float, audio: np.ndarray) -> torch.Tensor:
return torch.tensor(
[[float(audio[-1]), float(frame), float(len(audio))]],
dtype=torch.float32,
)
def commit_state(self, *, frame: float, audio: np.ndarray) -> torch.Tensor:
self.committed_until += len(audio) / 16_000.0
return self._state(frame, audio)
def temporary_state(self, *, frame: float, audio: np.ndarray) -> torch.Tensor:
before = self.committed_until
state = self._state(frame, audio)
if self.committed_until != before:
raise AssertionError("temporary state modified the fake main cache")
self.last_restore_audit = {
"kind": "snapshot_restore",
"llm_cache_length": 0,
"audio_cache_length": 0,
"audio_chunk_index": 0,
"streaming_position_offset": 0,
"window_pruning_suspended": True,
"window_policy_restored": True,
"llm_checksum_matches": True,
"audio_checksum_matches": True,
"history_matches": True,
"passed": True,
}
return state
class MiniCPMStreamingTests(unittest.TestCase):
def setUp(self) -> None:
self.config = MiniCPMStreamConfig(
boundary_max_abs=0.0,
boundary_min_cosine=1.0,
replay_max_abs=0.0,
replay_min_cosine=1.0,
)
def test_time_grid_is_right_edged_and_rejects_fractional_tail(self) -> None:
self.assertEqual(_time_grid(0.0, 1.0, 0.2), [0.2, 0.4, 0.6, 0.8, 1.0])
with self.assertRaisesRegex(ValueError, "integer number"):
_time_grid(0.0, 1.1, 0.2)
def test_native_hold_has_no_future_backfill(self) -> None:
extractor = MiniCPMFeatureExtractor(FakeReader(), FakeEngine(), self.config)
result = extractor.extract_native_hold(
start=0.0,
end=2.0,
show_progress=False,
)
self.assertEqual(tuple(result.states.shape), (10, 3))
self.assertTrue(torch.equal(result.states[:4], torch.zeros(4, 3)))
self.assertTrue(result.initialization_mask[:4].all())
self.assertFalse(result.initialization_mask[4:].any())
self.assertEqual(
result.native_update_mask.nonzero().flatten().tolist(),
[4, 9],
)
self.assertTrue(torch.equal(result.states[4], result.states[8]))
self.assertFalse(torch.equal(result.states[8], result.states[9]))
self.assertEqual(
result.state_source_times.tolist(),
[0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 2.0],
)
def test_cached_partial_updates_every_row_and_passes_audits(self) -> None:
reader = FakeReader()
extractor = MiniCPMFeatureExtractor(reader, FakeEngine(), self.config)
original_audio = reader.audio.copy()
result = extractor.extract_cached_partial(
start=0.0,
end=2.0,
verify_every=0,
future_intervention_every=0,
allow_unverified=False,
show_progress=False,
)
self.assertTrue(result.partial_tail_admissible)
self.assertTrue(np.array_equal(reader.audio, original_audio))
self.assertFalse(result.initialization_mask.any())
self.assertTrue(torch.equal(result.state_source_times, result.times))
self.assertEqual(
result.native_update_mask.nonzero().flatten().tolist(),
[4, 9],
)
self.assertEqual(
{item["kind"] for item in result.verification},
{
"native_boundary_equivalence",
"clean_stream_replay",
"future_audio_intervention",
"snapshot_restore",
},
)
self.assertTrue(all(item["passed"] for item in result.verification))
self.assertFalse(torch.equal(result.states[0], result.states[1]))
self.assertLess(float(result.states[0, 1]), 0.2)
def test_short_partial_run_is_rejected_without_boundary_audit(self) -> None:
extractor = MiniCPMFeatureExtractor(FakeReader(), FakeEngine(), self.config)
with self.assertRaisesRegex(RuntimeError, "missing=.*native_boundary"):
extractor.extract_cached_partial(
start=0.0,
end=0.8,
verify_every=0,
future_intervention_every=0,
allow_unverified=False,
show_progress=False,
)
def test_comparison_rejects_different_states(self) -> None:
item = _comparison(
torch.zeros(1, 2),
torch.ones(1, 2),
kind="test",
current_time=0.2,
max_abs_tolerance=0.0,
min_cosine=1.0,
)
self.assertFalse(item["passed"])
self.assertEqual(item["max_abs"], 1.0)
def test_transformers_version_is_explicitly_gated(self) -> None:
self.assertEqual(validate_transformers_version("4.51.0"), "4.51.0")
with self.assertRaisesRegex(RuntimeError, "found 4.52.0"):
validate_transformers_version("4.52.0")
self.assertEqual(
validate_transformers_version("4.52.0", allow_untested=True),
"4.52.0",
)
def test_saved_payload_loads_in_existing_head_and_unverified_fails(self) -> None:
states = torch.randn(20, 8)
payload = {
"format": FEATURE_FORMAT,
"states": states,
"times": (torch.arange(20, dtype=torch.float64) + 1) / 5,
"label_indices": torch.arange(20),
"mode": "causal-prefix",
"extractor_mode": "native-1hz-hold",
"config": {"step_seconds": 0.2},
}
targets = np.tile(
np.asarray(
[[1, 0, 0], [0, 1, 0], [0, 0, 1], [0, 1, 1]],
dtype=np.float32,
),
(5, 1),
)
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
feature_root = root / "features"
target_root = root / "targets"
feature_root.mkdir()
target_root.mkdir()
feature_path = feature_root / "Session_4.pt"
torch.save(payload, feature_path)
np.save(target_root / "Session_4.npy", targets)
loaded = load_session_data(feature_root, target_root, 4)
self.assertTrue(torch.equal(loaded.states, states))
times, indices = reference_alignment(payload)
self.assertTrue(torch.equal(times, payload["times"]))
self.assertTrue(torch.equal(indices, payload["label_indices"]))
bad_reference = dict(payload)
bad_reference["times"] = payload["times"] + 0.2
with self.assertRaisesRegex(ValueError, "Reference times"):
reference_alignment(bad_reference)
payload["mode"] = "minicpmo-unverified-partial-tail"
torch.save(payload, feature_path)
with self.assertRaisesRegex(ValueError, "expected 'causal-prefix'"):
load_session_data(feature_root, target_root, 4)
if __name__ == "__main__":
unittest.main()