Ego4D AV LLM-assisted floor-event annotation

This workflow mirrors the EasyCom boundary-review pipeline while using the official Ego4D AV annotation schema. It creates semantic review data for wearer floor entry and interruption-driven yield; it does not modify Ego4D's voice activity annotations and does not turn LLM output into ground truth.

Data and claim boundary

Expected dataset layout:

DATA_ROOT/
  v1/
    annotations/av_train.json
    annotations/av_val.json
    clips/<clip_uid>.mp4

The generator reads clip-local voice_segments, transcriptions, and social_segments_talking.is_at_me. The social label is annotation-only evidence. It must never be passed to a forecasting model or used after a strictly causal prediction cutoff. Ego4D's voice annotation guideline joins sounds separated by less than one second, so a stored segment boundary is an activity boundary, not automatically a semantic turn boundary.

The output split remains the split declared by Ego4D (train or val). Do not rename validation data to test or tune annotation rules after inspecting the final evaluation set.

1. Generate deterministic candidates

Start with one clip and inspect the JSONL before running the full set:

python3 prepare_ego4d_av_eot_int_review.py \
  --annotations "$DATA_ROOT/v1/annotations/av_train.json" \
  --output reports/ego4d_av_event_candidates/ego4d_av_train_candidates.jsonl \
  --limit 1

Then generate the selected splits:

python3 prepare_ego4d_av_eot_int_review.py \
  --annotations \
    "$DATA_ROOT/v1/annotations/av_train.json" \
    "$DATA_ROOT/v1/annotations/av_val.json" \
  --output reports/ego4d_av_event_candidates/ego4d_av_candidates.jsonl

The command also writes <output_stem>_timelines.json beside the JSONL. Each raw wearer onset and offset is retained. Same-other pauses and other-to-other handoffs are added as independently reviewable hard-negative proposals. Timing rules only select and describe candidates; they never assign semantic positive labels.

Official zero-duration voice records cannot define an activity interval. They are never expanded to an invented duration: the generator skips them, reports the count on stdout, and stores their complete source records under clip_audits and each candidate's temporal_evidence. A negative-duration record remains a hard error.

Ego4D v1 also contains nine train clips whose official voice endpoint extends past clip_end_sec by 0.42--1.03 seconds. The default audited ceiling is 1.1 seconds. These endpoints are preserved and flagged rather than silently clamped; larger overruns remain errors. A human should normally exclude a semantic boundary beyond observable media as DATA_RECORDING_BOUNDARY.

2. Inspect what will be uploaded

The LLM client refuses to upload transcript excerpts without an explicit opt-in. First inspect the exact prompt locally:

python3 adjudicate_ego4d_av_eot_int_candidates.py \
  --input reports/ego4d_av_event_candidates/ego4d_av_candidates.jsonl \
  --output reports/ego4d_av_event_candidates/ego4d_av_candidates_ai.jsonl \
  --max-events 8 \
  --dry-run

The prompt uses only timestamps, participant roles, provided transcripts, and provided is_at_me facts. It does not upload video or audio.

3. Run LLM pre-labeling

After confirming that external transcript upload is authorized:

export DEEPSEEK_API_KEY=...
python3 adjudicate_ego4d_av_eot_int_candidates.py \
  --input reports/ego4d_av_event_candidates/ego4d_av_candidates.jsonl \
  --output reports/ego4d_av_event_candidates/ego4d_av_candidates_ai.jsonl \
  --passes 2 \
  --batch-size 8 \
  --allow-external-transcript-upload

The client checkpoints after every successful batch. Two passes must agree; disagreement becomes EXCLUDE/INSUFFICIENT_EVIDENCE. Malformed singleton responses are written to a pending-human JSONL rather than silently dropped. Every successful LLM decision has needs_human_review=true.

4. Review in the browser

python3 review_ego4d_av_events.py \
  --events reports/ego4d_av_event_candidates/ego4d_av_candidates_ai.jsonl \
  --dataset-root "$DATA_ROOT" \
  --host 127.0.0.1 \
  --port 8765

Open http://127.0.0.1:8765. The UI shows the local video, clip-level speaker timeline, participant-aligned transcripts, LLM rationale, and the closed human label taxonomy. Reviews are atomically saved to reports/ego4d_av_event_reviews/reviews.jsonl with a CSV companion and can be resumed. The server rejects media paths outside DATA_ROOT and supports byte ranges for browser seeking.

For an SSH host, keep the service on loopback and forward the port:

ssh -L 8765:127.0.0.1:8765 USER@HOST

5. Freeze and audit

Before using these annotations in a paper or evaluator:

Freeze candidate windows and the closed taxonomy on training data.
Double-review a random sample plus every low-confidence/disagreement item.
Report raw candidates, positives, hard negatives, exclusions, and exclusion reasons separately for train and validation.
Print and independently inspect example timelines for entry, yield, backchannel, same-speaker pause, and other-to-other cases.
Keep the final label named human-adjudicated semantic annotation. LLM output alone is never ground truth.

Tests

python3 -m unittest -v \
  test_prepare_ego4d_av_eot_int_review.py \
  test_review_floor_events.py \
  test_adjudicate_easycom_eot_int_candidates.py

5.4 KiB Raw Blame History