205 lines
7.4 KiB
Markdown
205 lines
7.4 KiB
Markdown
# MiniCPM-o 4.5 causal global features
|
|
|
|
`streaming_minicpmo_features_directly.py` uses MiniCPM-o 4.5 only as a frozen
|
|
audio--visual feature extractor. It never calls `streaming_generate`, never
|
|
loads TTS, and never lets model-generated speech/listen tokens enter the
|
|
conversation history.
|
|
|
|
## Frozen modes
|
|
|
|
### `native-1hz-hold`
|
|
|
|
- The model commits one input-only AV chunk per completed second.
|
|
- The newest completed state is forward-held onto the 5 Hz target grid.
|
|
- The first four rows (0.2--0.8 s) use an all-zero initialization. The 1.0-s
|
|
state is never backfilled into those rows.
|
|
- `state_source_times`, `native_update_mask`, and `initialization_mask` make
|
|
this alignment auditable.
|
|
|
|
### `cached-partial-5hz`
|
|
|
|
- Completed one-second chunks remain in the official MiniCPM streaming cache.
|
|
- At a non-boundary 200-ms cutoff, the extractor snapshots the LLM cache,
|
|
audio-encoder cache, streaming Mel buffer, chunk index, and RNG state through
|
|
MiniCPM's official speculative-state API.
|
|
- It appends only the current half-open AV tail, reads the final normalized
|
|
Qwen3 hidden state, and restores the snapshot. Temporary closure never enters
|
|
the main cache.
|
|
- Native commits retain MiniCPM's normal streaming-window policy. Destructive
|
|
round pruning is suspended only inside the temporary transaction because the
|
|
upstream snapshot can truncate appended KV tokens but cannot recreate a
|
|
previously pruned round.
|
|
- At a one-second boundary, a complete native chunk is committed.
|
|
|
|
The 5 Hz branch is saved as the normal `mode=causal-prefix` contract only after
|
|
all mandatory checks pass:
|
|
|
|
1. a temporary full chunk matches the native committed state at a one-second
|
|
boundary;
|
|
2. a clean stream replay matches the cached result;
|
|
3. changing audio strictly after the cutoff leaves the current state exactly
|
|
unchanged;
|
|
4. snapshot restoration leaves LLM/audio cache lengths and checksums, chunk
|
|
history, position offset, and the audio chunk index unchanged.
|
|
|
|
With `--allow-unverified-partial-tail`, a failed or too-short audit may be
|
|
saved only as `mode=minicpmo-unverified-partial-tail`; the default head loader
|
|
rejects that artifact.
|
|
|
|
## Environment
|
|
|
|
The inspected upstream revision requires Python 3.10 and
|
|
`transformers==4.51.0`. Use a dedicated environment rather than changing the
|
|
working Qwen environment.
|
|
|
|
```bash
|
|
python -m pip install \
|
|
"transformers==4.51.0" \
|
|
accelerate \
|
|
"torch>=2.3.0,<=2.8.0" \
|
|
"torchaudio<=2.8.0" \
|
|
"minicpmo-utils>=1.0.5" \
|
|
"qwen-omni-utils[decord]"
|
|
```
|
|
|
|
The commands below use the upstream revision inspected during implementation,
|
|
`1f761131fa83f5ed3cd6f2f22b225c4501d154fa`. If the model is already stored
|
|
locally, replace `--model-id` with that snapshot directory and record its
|
|
immutable revision separately.
|
|
|
|
## Required smoke sequence
|
|
|
|
First validate the source timeline and decoder without loading MiniCPM:
|
|
|
|
```bash
|
|
python streaming_minicpmo_features_directly.py \
|
|
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
|
|
--end 3 \
|
|
--mode native-1hz-hold \
|
|
--decoder decord \
|
|
--media-only
|
|
```
|
|
|
|
Then extract three seconds with the native held-state baseline:
|
|
|
|
```bash
|
|
CUDA_VISIBLE_DEVICES=0 python streaming_minicpmo_features_directly.py \
|
|
--model-id openbmb/MiniCPM-o-4_5 \
|
|
--revision 1f761131fa83f5ed3cd6f2f22b225c4501d154fa \
|
|
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
|
|
--end 3 \
|
|
--mode native-1hz-hold \
|
|
--dtype bfloat16 \
|
|
--attention sdpa \
|
|
--device cuda:0 \
|
|
--decoder decord \
|
|
--decoder-threads 1 \
|
|
--max-open-readers 1 \
|
|
--output /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/smoke/Session_4_native_3s.pt
|
|
```
|
|
|
|
Run the verified cached partial-tail smoke test separately:
|
|
|
|
```bash
|
|
CUDA_VISIBLE_DEVICES=0 python streaming_minicpmo_features_directly.py \
|
|
--model-id openbmb/MiniCPM-o-4_5 \
|
|
--revision 1f761131fa83f5ed3cd6f2f22b225c4501d154fa \
|
|
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
|
|
--end 3 \
|
|
--mode cached-partial-5hz \
|
|
--verify-every 1 \
|
|
--future-intervention-every 1 \
|
|
--dtype bfloat16 \
|
|
--attention sdpa \
|
|
--device cuda:0 \
|
|
--decoder decord \
|
|
--decoder-threads 1 \
|
|
--max-open-readers 1 \
|
|
--output /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/smoke/Session_4_partial_3s.pt
|
|
```
|
|
|
|
Do not add `--allow-unverified-partial-tail` to a paper run. A verification
|
|
failure is a protocol failure to diagnose, not a tolerance to bypass.
|
|
|
|
Inspect both artifacts before a full session:
|
|
|
|
```bash
|
|
python - <<'PY'
|
|
from pathlib import Path
|
|
import torch
|
|
|
|
root = Path("/data1/yfzx/wangx/Datasets/easycom/minicpmo_features/smoke")
|
|
for name in ("Session_4_native_3s.pt", "Session_4_partial_3s.pt"):
|
|
payload = torch.load(root / name, map_location="cpu", weights_only=False)
|
|
print(name)
|
|
print(" mode:", payload["mode"], payload["extractor_mode"])
|
|
print(" states:", tuple(payload["states"].shape))
|
|
print(" times:", payload["times"][:6].tolist())
|
|
print(" source times:", payload["state_source_times"][:6].tolist())
|
|
print(" verified:", payload["partial_tail_admissible"])
|
|
print(" checks:", payload["verification"])
|
|
PY
|
|
```
|
|
|
|
Expected alignment:
|
|
|
|
```text
|
|
native times: 0.2 0.4 0.6 0.8 1.0 1.2 ...
|
|
native source time: 0.0 0.0 0.0 0.0 1.0 1.0 ...
|
|
partial source time: 0.2 0.4 0.6 0.8 1.0 1.2 ...
|
|
```
|
|
|
|
## Full Session 4
|
|
|
|
Native 1 Hz held features:
|
|
|
|
```bash
|
|
mkdir -p /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/native_1hz_hold
|
|
|
|
CUDA_VISIBLE_DEVICES=0 python streaming_minicpmo_features_directly.py \
|
|
--model-id openbmb/MiniCPM-o-4_5 \
|
|
--revision 1f761131fa83f5ed3cd6f2f22b225c4501d154fa \
|
|
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
|
|
--target /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame/Session_4.npy \
|
|
--mode native-1hz-hold \
|
|
--dtype bfloat16 \
|
|
--attention sdpa \
|
|
--device cuda:0 \
|
|
--decoder decord \
|
|
--decoder-threads 1 \
|
|
--max-open-readers 1 \
|
|
--frame-cache-size 32 \
|
|
--output /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/native_1hz_hold/Session_4.pt
|
|
```
|
|
|
|
Verified 5 Hz cached partial-tail features:
|
|
|
|
```bash
|
|
mkdir -p /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/cached_partial_5hz
|
|
|
|
CUDA_VISIBLE_DEVICES=0 python streaming_minicpmo_features_directly.py \
|
|
--model-id openbmb/MiniCPM-o-4_5 \
|
|
--revision 1f761131fa83f5ed3cd6f2f22b225c4501d154fa \
|
|
--easycom-session-dir /data1/yfzx/wangx/Datasets/easycom/extracted_easycom/Main/Video_Compressed/Session_4 \
|
|
--target /data1/yfzx/wangx/Datasets/easycom/easycom_target_frame/Session_4.npy \
|
|
--mode cached-partial-5hz \
|
|
--dtype bfloat16 \
|
|
--attention sdpa \
|
|
--device cuda:0 \
|
|
--decoder decord \
|
|
--decoder-threads 1 \
|
|
--max-open-readers 1 \
|
|
--frame-cache-size 32 \
|
|
--output /data1/yfzx/wangx/Datasets/easycom/minicpmo_features/cached_partial_5hz/Session_4.pt
|
|
```
|
|
|
|
The full run still executes the first clean-replay and future-intervention
|
|
checks plus every one-second boundary check. Add `--verify-every 250` and
|
|
`--future-intervention-every 250` for periodic 50-second audits; smaller values
|
|
make long sessions increasingly expensive because a clean audit replays the
|
|
prefix from the session start.
|
|
|
|
After extraction, either directory can be passed directly as
|
|
`train_turn_head.py --feature-root ...` because verified artifacts retain the
|
|
existing dense global-feature contract.
|