ZNJJ-api-server/.agents/skills/prompt-engineer/references/prompt-optimization.md

# Prompt Optimization

---

## The Optimization Loop

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                         PROMPT OPTIMIZATION CYCLE                           │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│    ┌──────────┐     ┌──────────┐     ┌──────────┐     ┌──────────┐        │
│    │ Baseline │────▶│  Measure │────▶│ Diagnose │────▶│  Change  │        │
│    │  Prompt  │     │ Results  │     │  Issues  │     │   One    │        │
│    └──────────┘     └──────────┘     └──────────┘     └────┬─────┘        │
│         ▲                                                   │              │
│         │                                                   │              │
│         └───────────────────────────────────────────────────┘              │
│                           (Iterate until target met)                       │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘
```

> **CRITICAL RULE: Change one variable at a time.** Multiple simultaneous changes make it impossible to identify what worked.

---

## Establishing a Baseline

Before optimizing, establish clear metrics and baseline performance.

### Baseline Checklist

```markdown
## Prompt Baseline Document

### Prompt Version: v1.0.0
### Date: YYYY-MM-DD
### Model: claude-opus-4-5-20251101

### Task Definition
[What should the prompt accomplish?]

### Success Criteria
- Primary metric: [e.g., accuracy >= 95%]
- Secondary metrics: [e.g., latency < 2s, cost < $0.01/request]

### Test Set
- Size: [number of test cases]
- Source: [how test cases were collected]
- Categories: [breakdown by type/difficulty]

### Baseline Results
| Metric | Value | Target |
|--------|-------|--------|
| Accuracy | 82% | 95% |
| Avg latency | 1.8s | <2s |
| Avg tokens | 450 | <300 |
| Cost/request | $0.015 | <$0.01 |
```

### Creating a Representative Test Set

```python
def create_test_set(task_type: str, size: int = 100) -> list:
    """Create a diverse test set for prompt evaluation."""
    test_cases = []

    # Include different categories
    categories = {
        "typical": 0.60,      # Common cases (60%)
        "edge_case": 0.20,    # Boundary conditions (20%)
        "adversarial": 0.10,  # Tricky inputs (10%)
        "malformed": 0.10,    # Invalid/unusual inputs (10%)
    }

    for category, proportion in categories.items():
        count = int(size * proportion)
        test_cases.extend(generate_cases(task_type, category, count))

    return test_cases
```

---

## Diagnostic Framework

When prompts underperform, diagnose the root cause before changing anything.

### Failure Category Analysis

| Failure Type | Symptoms | Common Causes |
|--------------|----------|---------------|
| **Format errors** | Wrong structure, missing fields | Unclear format spec, no examples |
| **Hallucinations** | Made-up facts, wrong answers | Lack of grounding, vague instructions |
| **Inconsistency** | Same input, different outputs | Ambiguous instructions, high temperature |
| **Over-verbosity** | Too much explanation | No length constraints, wrong audience |
| **Under-performance** | Low accuracy across board | Wrong pattern choice, insufficient context |
| **Edge case failures** | Breaks on unusual inputs | Missing constraint handling |

### Diagnostic Questions

```markdown
## Prompt Diagnostic Checklist

### 1. Instruction Clarity
- [ ] Is the task unambiguously defined?
- [ ] Are constraints explicit?
- [ ] Is the output format specified?

### 2. Context Sufficiency
- [ ] Does the model have all needed information?
- [ ] Are examples representative of real inputs?
- [ ] Is domain knowledge assumed correctly?

### 3. Edge Case Coverage
- [ ] Empty inputs?
- [ ] Maximum length inputs?
- [ ] Invalid/malformed inputs?
- [ ] Ambiguous cases?

### 4. Instruction Conflicts
- [ ] Do any instructions contradict each other?
- [ ] Do examples match the instructions?
- [ ] Are constraints achievable together?
```

### Error Analysis Template

```python
def analyze_failures(results: list) -> dict:
    """Categorize and analyze prompt failures."""
    analysis = {
        "total": len(results),
        "passed": 0,
        "failed": 0,
        "failure_categories": {},
        "examples": []
    }

    for result in results:
        if result["passed"]:
            analysis["passed"] += 1
        else:
            analysis["failed"] += 1
            category = categorize_failure(result)
            analysis["failure_categories"][category] = \
                analysis["failure_categories"].get(category, 0) + 1

            # Keep first 3 examples per category
            if len([e for e in analysis["examples"] if e["category"] == category]) < 3:
                analysis["examples"].append({
                    "category": category,
                    "input": result["input"],
                    "expected": result["expected"],
                    "actual": result["actual"],
                    "hypothesis": generate_hypothesis(result)
                })

    return analysis
```

---

## Optimization Techniques

### Technique 1: Instruction Refinement

**Problem:** Vague or ambiguous instructions leading to inconsistent outputs.

**Before:**
```
Summarize this article.

{article}
```

**After:**
```
Summarize the following article in exactly 2-3 sentences.
Focus on the main conclusion and key supporting evidence.
Do not include quotes or specific numbers unless essential.
Write for a general audience with no assumed domain knowledge.

Article:
{article}

Summary:
```

### Technique 2: Constraint Tightening

**Problem:** Outputs that are technically correct but don't meet practical needs.

**Before:**
```
Extract the email addresses from this text.

{text}
```

**After:**
```
Extract all valid email addresses from the following text.

Requirements:
- Return as a JSON array of strings
- Return empty array [] if no emails found
- Only include properly formatted emails (user@domain.tld)
- Deduplicate - each email appears once
- Sort alphabetically

Text:
{text}

Emails:
```

### Technique 3: Example Calibration

**Problem:** Few-shot examples that don't match real-world input distribution.

```python
def calibrate_examples(example_pool: list, real_inputs: list, k: int = 5) -> list:
    """Select examples that match the distribution of real inputs."""
    # Cluster real inputs
    real_clusters = cluster_by_embedding(real_inputs, n_clusters=k)

    # For each cluster, find best matching example
    calibrated = []
    for cluster_center in real_clusters:
        best_match = max(
            example_pool,
            key=lambda ex: cosine_similarity(embed(ex["input"]), cluster_center)
        )
        calibrated.append(best_match)

    return calibrated
```

### Technique 4: Output Scaffolding

**Problem:** Model produces correct content but wrong structure.

**Before:**
```
Analyze this code for security issues.
```

**After:**
```
Analyze this code for security issues using the following structure:

## Summary
[One sentence overview]

## Issues Found
For each issue:
- **Severity:** [Critical/High/Medium/Low]
- **Location:** [file:line or function name]
- **Description:** [What's wrong]
- **Fix:** [How to remediate]

## Recommendation
[Overall assessment and priority order for fixes]

Code:
{code}
```

---

## Token Optimization

### Token Reduction Strategies

| Strategy | Savings | Risk | When to Use |
|----------|---------|------|-------------|
| Remove redundant instructions | 10-20% | Low | Always |
| Shorten examples | 20-40% | Medium | Token-constrained |
| Use abbreviations/symbols | 5-15% | Medium | Technical audiences |
| Compress context | 30-50% | High | Very long inputs |
| Switch to zero-shot | 40-60% | High | Simple tasks |

### Before/After: Token Reduction

**Before (180 tokens):**
```
You are a helpful assistant that specializes in analyzing customer feedback
and extracting sentiment information. Your task is to read the customer
review provided below and determine whether the overall sentiment expressed
in the review is positive, negative, or neutral. Please respond with exactly
one word: either "positive", "negative", or "neutral". Do not include any
other text, explanations, or formatting in your response.

Customer Review:
{review}

Sentiment:
```

**After (45 tokens):**
```
Classify sentiment as: positive, negative, or neutral.
Reply with one word only.

Review: {review}

Sentiment:
```

### Measuring Token Impact

```python
import tiktoken

def compare_token_usage(prompt_v1: str, prompt_v2: str, model: str = "gpt-4") -> dict:
    """Compare token usage between two prompt versions."""
    enc = tiktoken.encoding_for_model(model)

    v1_tokens = len(enc.encode(prompt_v1))
    v2_tokens = len(enc.encode(prompt_v2))

    return {
        "v1_tokens": v1_tokens,
        "v2_tokens": v2_tokens,
        "difference": v1_tokens - v2_tokens,
        "reduction_pct": ((v1_tokens - v2_tokens) / v1_tokens) * 100,
        "cost_impact": estimate_cost_savings(v1_tokens, v2_tokens, model)
    }
```

### Context Compression Techniques

```python
def compress_context(text: str, target_ratio: float = 0.5) -> str:
    """Compress context while preserving key information."""

    # Strategy 1: Extractive summarization
    key_sentences = extract_key_sentences(text, ratio=target_ratio)

    # Strategy 2: Remove redundancy
    deduplicated = remove_redundant_info(key_sentences)

    # Strategy 3: Use LLM for compression
    compressed = llm.complete(f"""
    Compress the following text to {int(target_ratio * 100)}% of its length.
    Preserve all facts, numbers, and key details.
    Remove only redundant or low-information content.

    Text: {deduplicated}

    Compressed:
    """)

    return compressed
```

---

## A/B Testing Framework

### Test Design

```python
class PromptABTest:
    """Framework for A/B testing prompt variants."""

    def __init__(self, prompt_a: str, prompt_b: str, test_cases: list):
        self.prompt_a = prompt_a
        self.prompt_b = prompt_b
        self.test_cases = test_cases
        self.results = {"a": [], "b": []}

    def run(self, sample_size: int = 100) -> dict:
        """Run A/B test with randomized assignment."""
        import random

        for test_case in random.sample(self.test_cases, sample_size):
            # Randomize order to avoid position bias
            if random.random() < 0.5:
                result_a = self.evaluate(self.prompt_a, test_case)
                result_b = self.evaluate(self.prompt_b, test_case)
            else:
                result_b = self.evaluate(self.prompt_b, test_case)
                result_a = self.evaluate(self.prompt_a, test_case)

            self.results["a"].append(result_a)
            self.results["b"].append(result_b)

        return self.analyze_results()

    def analyze_results(self) -> dict:
        """Statistical analysis of A/B test results."""
        from scipy import stats

        scores_a = [r["score"] for r in self.results["a"]]
        scores_b = [r["score"] for r in self.results["b"]]

        t_stat, p_value = stats.ttest_ind(scores_a, scores_b)

        return {
            "prompt_a_mean": sum(scores_a) / len(scores_a),
            "prompt_b_mean": sum(scores_b) / len(scores_b),
            "p_value": p_value,
            "significant": p_value < 0.05,
            "winner": "a" if sum(scores_a) > sum(scores_b) else "b",
            "confidence": 1 - p_value
        }
```

### Minimum Sample Size Calculation

```python
def calculate_sample_size(
    baseline_rate: float,
    minimum_detectable_effect: float,
    significance_level: float = 0.05,
    power: float = 0.80
) -> int:
    """Calculate required sample size for detecting a given effect."""
    from scipy import stats

    # Effect size (Cohen's h for proportions)
    p1 = baseline_rate
    p2 = baseline_rate + minimum_detectable_effect
    h = 2 * (math.asin(math.sqrt(p1)) - math.asin(math.sqrt(p2)))

    # Required sample size per group
    z_alpha = stats.norm.ppf(1 - significance_level / 2)
    z_beta = stats.norm.ppf(power)

    n = 2 * ((z_alpha + z_beta) / h) ** 2

    return math.ceil(n)

# Example: Detect 5% improvement from 80% baseline
# sample_size = calculate_sample_size(0.80, 0.05)  # ~783 per group
```

---

## Version Control for Prompts

### Prompt Versioning Schema

```yaml
# prompt_registry.yaml
prompts:
  sentiment_classifier:
    current: v2.1.0
    versions:
      v1.0.0:
        file: prompts/sentiment/v1.0.0.txt
        date: 2024-01-15
        metrics:
          accuracy: 0.82
          latency_p50: 1.2s
        status: deprecated

      v2.0.0:
        file: prompts/sentiment/v2.0.0.txt
        date: 2024-02-01
        metrics:
          accuracy: 0.89
          latency_p50: 1.1s
        changes:
          - Added few-shot examples
          - Tightened output format
        status: deprecated

      v2.1.0:
        file: prompts/sentiment/v2.1.0.txt
        date: 2024-02-15
        metrics:
          accuracy: 0.94
          latency_p50: 1.0s
        changes:
          - Optimized examples for edge cases
          - Reduced token count by 30%
        status: production
```

### Change Documentation Template

```markdown
## Prompt Change Record

### Version: v2.0.0 -> v2.1.0
### Date: 2024-02-15
### Author: [name]

### Problem Statement
Accuracy dropped to 85% on sarcastic reviews (edge case category).

### Hypothesis
Current examples don't include sarcastic tone, causing misclassification.

### Changes Made
1. Added 2 sarcastic review examples
2. Added instruction: "Consider tone and context, not just words"
3. Removed verbose instruction paragraph (token optimization)

### Test Results
| Metric | v2.0.0 | v2.1.0 | Change |
|--------|--------|--------|--------|
| Overall accuracy | 89% | 94% | +5% |
| Sarcasm accuracy | 62% | 91% | +29% |
| Tokens | 156 | 109 | -30% |

### Rollback Plan
Revert to v2.0.0 if accuracy drops below 90% in production.
```

---

## Common Optimization Mistakes

| Mistake | Why It's Wrong | Better Approach |
|---------|----------------|-----------------|
| Multiple changes at once | Can't identify what worked | One change per iteration |
| Testing on training examples | Overfitting to test set | Hold out validation set |
| Optimizing for edge cases first | May hurt common case | Fix common cases first |
| Ignoring latency/cost | Production constraints matter | Track all metrics |
| No baseline measurement | Can't prove improvement | Always measure first |
| Skipping failure analysis | Symptoms vs. root cause | Diagnose before changing |

---

## Optimization Decision Tree

```
                    ┌──────────────────────────┐
                    │   Prompt Underperforms   │
                    └────────────┬─────────────┘
                                 │
                    ┌────────────▼─────────────┐
                    │  What's the failure mode? │
                    └────────────┬─────────────┘
                                 │
        ┌────────────────────────┼────────────────────────┐
        │                        │                        │
  Format Issues            Wrong Content           Inconsistent
        │                        │                        │
        ▼                        ▼                        ▼
┌───────────────┐      ┌───────────────┐      ┌───────────────┐
│ Add output    │      │ Improve       │      │ Add examples  │
│ scaffolding   │      │ instructions  │      │ Lower temp    │
│ Add examples  │      │ Add context   │      │ Add constraints│
└───────────────┘      │ Use CoT       │      └───────────────┘
                       └───────────────┘
```

---

## Related Skills

- **Evaluation Frameworks** - Measuring prompt performance systematically
- **Fine-Tuning Expert** - When optimization hits limits
- **Cost Engineer** - Token and latency optimization at scale