icassp2027-experiment/docs/ego4d_av_llm_annotation_review.md

# Ego4D AV LLM-assisted floor-event annotation

This workflow mirrors the EasyCom boundary-review pipeline while using the
official Ego4D AV annotation schema. It creates semantic review data for
wearer floor entry and interruption-driven yield; it does not modify Ego4D's
voice activity annotations and does not turn LLM output into ground truth.

## Data and claim boundary

Expected dataset layout:

```text
DATA_ROOT/
  v1/
    annotations/av_train.json
    annotations/av_val.json
    clips/<clip_uid>.mp4
```

The generator reads clip-local `voice_segments`, `transcriptions`, and
`social_segments_talking.is_at_me`. The social label is annotation-only
evidence. It must never be passed to a forecasting model or used after a
strictly causal prediction cutoff. Ego4D's voice annotation guideline joins
sounds separated by less than one second, so a stored segment boundary is an
activity boundary, not automatically a semantic turn boundary.

The output split remains the split declared by Ego4D (`train` or `val`). Do not
rename validation data to test or tune annotation rules after inspecting the
final evaluation set.

## 1. Generate deterministic candidates

Start with one clip and inspect the JSONL before running the full set:

```bash
python3 prepare_ego4d_av_eot_int_review.py \
  --annotations "$DATA_ROOT/v1/annotations/av_train.json" \
  --output reports/ego4d_av_event_candidates/ego4d_av_train_candidates.jsonl \
  --limit 1
```

Then generate the selected splits:

```bash
python3 prepare_ego4d_av_eot_int_review.py \
  --annotations \
    "$DATA_ROOT/v1/annotations/av_train.json" \
    "$DATA_ROOT/v1/annotations/av_val.json" \
  --output reports/ego4d_av_event_candidates/ego4d_av_candidates.jsonl
```

The command also writes `<output_stem>_timelines.json` beside the JSONL. Each
raw wearer onset and offset is retained. Same-other pauses and other-to-other
handoffs are added as independently reviewable hard-negative proposals.
Timing rules only select and describe candidates; they never assign semantic
positive labels.

Official zero-duration voice records cannot define an activity interval. They
are never expanded to an invented duration: the generator skips them, reports
the count on stdout, and stores their complete source records under
`clip_audits` and each candidate's `temporal_evidence`. A negative-duration
record remains a hard error.

Ego4D v1 also contains nine train clips whose official voice endpoint extends
past `clip_end_sec` by 0.42--1.03 seconds. The default audited ceiling is 1.1
seconds. These endpoints are preserved and flagged rather than silently
clamped; larger overruns remain errors. A human should normally exclude a
semantic boundary beyond observable media as `DATA_RECORDING_BOUNDARY`.

## 2. Inspect what will be uploaded

The LLM client refuses to upload transcript excerpts without an explicit
opt-in. First inspect the exact prompt locally:

```bash
python3 adjudicate_ego4d_av_eot_int_candidates.py \
  --input reports/ego4d_av_event_candidates/ego4d_av_candidates.jsonl \
  --output reports/ego4d_av_event_candidates/ego4d_av_candidates_ai.jsonl \
  --max-events 8 \
  --dry-run
```

The prompt uses only timestamps, participant roles, provided transcripts, and
provided `is_at_me` facts. It does not upload video or audio.

## 3. Run LLM pre-labeling

After confirming that external transcript upload is authorized:

```bash
export DEEPSEEK_API_KEY=...
python3 adjudicate_ego4d_av_eot_int_candidates.py \
  --input reports/ego4d_av_event_candidates/ego4d_av_candidates.jsonl \
  --output reports/ego4d_av_event_candidates/ego4d_av_candidates_ai.jsonl \
  --passes 2 \
  --batch-size 8 \
  --allow-external-transcript-upload
```

The client checkpoints after every successful batch. Two passes must agree;
disagreement becomes `EXCLUDE/INSUFFICIENT_EVIDENCE`. Malformed singleton
responses are written to a pending-human JSONL rather than silently dropped.
Every successful LLM decision has `needs_human_review=true`.

## 4. Review in the browser

```bash
python3 review_ego4d_av_events.py \
  --events reports/ego4d_av_event_candidates/ego4d_av_candidates_ai.jsonl \
  --dataset-root "$DATA_ROOT" \
  --host 127.0.0.1 \
  --port 8765
```

Open `http://127.0.0.1:8765`. The UI shows the local video, clip-level speaker
timeline, participant-aligned transcripts, LLM rationale, and the closed human
label taxonomy. Reviews are atomically saved to
`reports/ego4d_av_event_reviews/reviews.jsonl` with a CSV companion and can be
resumed. The server rejects media paths outside `DATA_ROOT` and supports byte
ranges for browser seeking.

For an SSH host, keep the service on loopback and forward the port:

```bash
ssh -L 8765:127.0.0.1:8765 USER@HOST
```

## 5. Freeze and audit

Before using these annotations in a paper or evaluator:

1. Freeze candidate windows and the closed taxonomy on training data.
2. Double-review a random sample plus every low-confidence/disagreement item.
3. Report raw candidates, positives, hard negatives, exclusions, and exclusion
   reasons separately for train and validation.
4. Print and independently inspect example timelines for entry, yield,
   backchannel, same-speaker pause, and other-to-other cases.
5. Keep the final label named human-adjudicated semantic annotation. LLM output
   alone is never ground truth.

## Tests

```bash
python3 -m unittest -v \
  test_prepare_ego4d_av_eot_int_review.py \
  test_review_floor_events.py \
  test_adjudicate_easycom_eot_int_candidates.py
```