chore: track shared agent skills in repository
This commit is contained in:
Eric Wang
@@ -0,0 +1,447 @@
|
||||
---
|
||||
name: promptfoo-evaluation
|
||||
description: Configures and runs LLM evaluation using Promptfoo framework. Use when setting up prompt testing, creating evaluation configs (promptfooconfig.yaml), writing Python custom assertions, implementing llm-rubric for LLM-as-judge, or managing few-shot examples in prompts. Triggers on keywords like "promptfoo", "eval", "LLM evaluation", "prompt testing", or "model comparison".
|
||||
---
|
||||
|
||||
# Promptfoo Evaluation
|
||||
|
||||
## Overview
|
||||
|
||||
This skill provides guidance for configuring and running LLM evaluations using [Promptfoo](https://www.promptfoo.dev/), an open-source CLI tool for testing and comparing LLM outputs.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Initialize a new evaluation project
|
||||
npx promptfoo@latest init
|
||||
|
||||
# Run evaluation
|
||||
npx promptfoo@latest eval
|
||||
|
||||
# View results in browser
|
||||
npx promptfoo@latest view
|
||||
```
|
||||
|
||||
## Configuration Structure
|
||||
|
||||
A typical Promptfoo project structure:
|
||||
|
||||
```
|
||||
project/
|
||||
├── promptfooconfig.yaml # Main configuration
|
||||
├── prompts/
|
||||
│ ├── system.md # System prompt
|
||||
│ └── chat.json # Chat format prompt
|
||||
├── tests/
|
||||
│ └── cases.yaml # Test cases
|
||||
└── scripts/
|
||||
└── metrics.py # Custom Python assertions
|
||||
```
|
||||
|
||||
## Core Configuration (promptfooconfig.yaml)
|
||||
|
||||
```yaml
|
||||
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
||||
description: "My LLM Evaluation"
|
||||
|
||||
# Prompts to test
|
||||
prompts:
|
||||
- file://prompts/system.md
|
||||
- file://prompts/chat.json
|
||||
|
||||
# Models to compare
|
||||
providers:
|
||||
- id: anthropic:messages:claude-sonnet-4-6
|
||||
label: Claude-Sonnet-4.6
|
||||
- id: openai:gpt-4.1
|
||||
label: GPT-4.1
|
||||
|
||||
# Test cases
|
||||
tests: file://tests/cases.yaml
|
||||
|
||||
# Concurrency control (MUST be under commandLineOptions, NOT top-level)
|
||||
commandLineOptions:
|
||||
maxConcurrency: 2
|
||||
|
||||
# Default assertions for all tests
|
||||
defaultTest:
|
||||
assert:
|
||||
- type: python
|
||||
value: file://scripts/metrics.py:custom_assert
|
||||
- type: llm-rubric
|
||||
value: |
|
||||
Evaluate the response quality on a 0-1 scale.
|
||||
threshold: 0.7
|
||||
|
||||
# Output path
|
||||
outputPath: results/eval-results.json
|
||||
```
|
||||
|
||||
## Prompt Formats
|
||||
|
||||
### Text Prompt (system.md)
|
||||
|
||||
```markdown
|
||||
You are a helpful assistant.
|
||||
|
||||
Task: {{task}}
|
||||
Context: {{context}}
|
||||
```
|
||||
|
||||
### Chat Format (chat.json)
|
||||
|
||||
```json
|
||||
[
|
||||
{"role": "system", "content": "{{system_prompt}}"},
|
||||
{"role": "user", "content": "{{user_input}}"}
|
||||
]
|
||||
```
|
||||
|
||||
### Few-Shot Pattern
|
||||
|
||||
Embed examples directly in prompt or use chat format with assistant messages:
|
||||
|
||||
```json
|
||||
[
|
||||
{"role": "system", "content": "{{system_prompt}}"},
|
||||
{"role": "user", "content": "Example input: {{example_input}}"},
|
||||
{"role": "assistant", "content": "{{example_output}}"},
|
||||
{"role": "user", "content": "Now process: {{actual_input}}"}
|
||||
]
|
||||
```
|
||||
|
||||
## Test Cases (tests/cases.yaml)
|
||||
|
||||
```yaml
|
||||
- description: "Test case 1"
|
||||
vars:
|
||||
system_prompt: file://prompts/system.md
|
||||
user_input: "Hello world"
|
||||
# Load content from files
|
||||
context: file://data/context.txt
|
||||
assert:
|
||||
- type: contains
|
||||
value: "expected text"
|
||||
- type: python
|
||||
value: file://scripts/metrics.py:custom_check
|
||||
threshold: 0.8
|
||||
```
|
||||
|
||||
## Python Custom Assertions
|
||||
|
||||
Create a Python file for custom assertions (e.g., `scripts/metrics.py`):
|
||||
|
||||
```python
|
||||
def get_assert(output: str, context: dict) -> dict:
|
||||
"""Default assertion function."""
|
||||
vars_dict = context.get('vars', {})
|
||||
|
||||
# Access test variables
|
||||
expected = vars_dict.get('expected', '')
|
||||
|
||||
# Return result
|
||||
return {
|
||||
"pass": expected in output,
|
||||
"score": 0.8,
|
||||
"reason": "Contains expected content",
|
||||
"named_scores": {"relevance": 0.9}
|
||||
}
|
||||
|
||||
def custom_check(output: str, context: dict) -> dict:
|
||||
"""Custom named assertion."""
|
||||
word_count = len(output.split())
|
||||
passed = 100 <= word_count <= 500
|
||||
|
||||
return {
|
||||
"pass": passed,
|
||||
"score": min(1.0, word_count / 300),
|
||||
"reason": f"Word count: {word_count}"
|
||||
}
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
- Default function name is `get_assert`
|
||||
- Specify function with `file://path.py:function_name`
|
||||
- Return `bool`, `float` (score), or `dict` with pass/score/reason
|
||||
- Access variables via `context['vars']`
|
||||
|
||||
## LLM-as-Judge (llm-rubric)
|
||||
|
||||
```yaml
|
||||
assert:
|
||||
- type: llm-rubric
|
||||
value: |
|
||||
Evaluate the response based on:
|
||||
1. Accuracy of information
|
||||
2. Clarity of explanation
|
||||
3. Completeness
|
||||
|
||||
Score 0.0-1.0 where 0.7+ is passing.
|
||||
threshold: 0.7
|
||||
provider: openai:gpt-4.1 # Optional: override grader model
|
||||
```
|
||||
|
||||
**When using a relay/proxy API**, each `llm-rubric` assertion needs its own `provider` config with `apiBaseUrl`. Otherwise the grader falls back to the default Anthropic/OpenAI endpoint and gets 401 errors:
|
||||
|
||||
```yaml
|
||||
assert:
|
||||
- type: llm-rubric
|
||||
value: |
|
||||
Evaluate quality on a 0-1 scale.
|
||||
threshold: 0.7
|
||||
provider:
|
||||
id: anthropic:messages:claude-sonnet-4-6
|
||||
config:
|
||||
apiBaseUrl: https://your-relay.example.com/api
|
||||
```
|
||||
|
||||
**Best practices:**
|
||||
- Provide clear scoring criteria
|
||||
- Use `threshold` to set minimum passing score
|
||||
- Default grader uses available API keys (OpenAI → Anthropic → Google)
|
||||
- **When using relay/proxy**: every `llm-rubric` must have its own `provider` with `apiBaseUrl` — the main provider's `apiBaseUrl` is NOT inherited
|
||||
|
||||
## Common Assertion Types
|
||||
|
||||
| Type | Usage | Example |
|
||||
|------|-------|---------|
|
||||
| `contains` | Check substring | `value: "hello"` |
|
||||
| `icontains` | Case-insensitive | `value: "HELLO"` |
|
||||
| `equals` | Exact match | `value: "42"` |
|
||||
| `regex` | Pattern match | `value: "\\d{4}"` |
|
||||
| `python` | Custom logic | `value: file://script.py` |
|
||||
| `llm-rubric` | LLM grading | `value: "Is professional"` |
|
||||
| `latency` | Response time | `threshold: 1000` |
|
||||
|
||||
## File References
|
||||
|
||||
All `file://` paths are resolved relative to `promptfooconfig.yaml` location (NOT the YAML file containing the reference). This is a common gotcha when `tests:` references a separate YAML file — the `file://` paths inside that test file still resolve from the config root.
|
||||
|
||||
```yaml
|
||||
# Load file content as variable
|
||||
vars:
|
||||
content: file://data/input.txt
|
||||
|
||||
# Load prompt from file
|
||||
prompts:
|
||||
- file://prompts/main.md
|
||||
|
||||
# Load test cases from file
|
||||
tests: file://tests/cases.yaml
|
||||
|
||||
# Load Python assertion
|
||||
assert:
|
||||
- type: python
|
||||
value: file://scripts/check.py:validate
|
||||
```
|
||||
|
||||
## Running Evaluations
|
||||
|
||||
```bash
|
||||
# Basic run
|
||||
npx promptfoo@latest eval
|
||||
|
||||
# With specific config
|
||||
npx promptfoo@latest eval --config path/to/config.yaml
|
||||
|
||||
# Output to file
|
||||
npx promptfoo@latest eval --output results.json
|
||||
|
||||
# Filter tests
|
||||
npx promptfoo@latest eval --filter-metadata category=math
|
||||
|
||||
# View results
|
||||
npx promptfoo@latest view
|
||||
```
|
||||
|
||||
## Relay / Proxy API Configuration
|
||||
|
||||
When using an API relay or proxy instead of direct Anthropic/OpenAI endpoints:
|
||||
|
||||
```yaml
|
||||
providers:
|
||||
- id: anthropic:messages:claude-sonnet-4-6
|
||||
label: Claude-Sonnet-4.6
|
||||
config:
|
||||
max_tokens: 4096
|
||||
apiBaseUrl: https://your-relay.example.com/api # Promptfoo appends /v1/messages
|
||||
|
||||
# CRITICAL: maxConcurrency MUST be under commandLineOptions (NOT top-level)
|
||||
commandLineOptions:
|
||||
maxConcurrency: 1 # Respect relay rate limits
|
||||
```
|
||||
|
||||
**Key rules:**
|
||||
- `apiBaseUrl` goes in `providers[].config` — Promptfoo appends `/v1/messages` automatically
|
||||
- `maxConcurrency` must be under `commandLineOptions:` — placing it at top level is silently ignored
|
||||
- When using relay with LLM-as-judge, set `maxConcurrency: 1` to avoid concurrent request limits (generation + grading share the same pool)
|
||||
- Pass relay token as `ANTHROPIC_API_KEY` env var
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Python not found:**
|
||||
```bash
|
||||
export PROMPTFOO_PYTHON=python3
|
||||
```
|
||||
|
||||
**Large outputs truncated:**
|
||||
Outputs over 30000 characters are truncated. Use `head_limit` in assertions.
|
||||
|
||||
**File not found errors:**
|
||||
All `file://` paths resolve relative to `promptfooconfig.yaml` location.
|
||||
|
||||
**maxConcurrency ignored (shows "up to N at a time"):**
|
||||
`maxConcurrency` must be under `commandLineOptions:`, not at the YAML top level. This is a common mistake.
|
||||
|
||||
**LLM-as-judge returns 401 with relay API:**
|
||||
Each `llm-rubric` assertion must have its own `provider` with `apiBaseUrl`. The main provider config is not inherited by grader assertions.
|
||||
|
||||
**HTML tags in model output inflating metrics:**
|
||||
Models may output `<br>`, `<b>`, etc. in structured content. Strip HTML in Python assertions before measuring:
|
||||
```python
|
||||
import re
|
||||
clean_text = re.sub(r'<[^>]+>', '', raw_text)
|
||||
```
|
||||
|
||||
## Echo Provider (Preview Mode)
|
||||
|
||||
Use the **echo provider** to preview rendered prompts without making API calls:
|
||||
|
||||
```yaml
|
||||
# promptfooconfig-preview.yaml
|
||||
providers:
|
||||
- echo # Returns prompt as output, no API calls
|
||||
|
||||
tests:
|
||||
- vars:
|
||||
input: "test content"
|
||||
```
|
||||
|
||||
**Use cases:**
|
||||
- Preview prompt rendering before expensive API calls
|
||||
- Verify Few-shot examples are loaded correctly
|
||||
- Debug variable substitution issues
|
||||
- Validate prompt structure
|
||||
|
||||
```bash
|
||||
# Run preview mode
|
||||
npx promptfoo@latest eval --config promptfooconfig-preview.yaml
|
||||
```
|
||||
|
||||
**Cost:** Free - no API tokens consumed.
|
||||
|
||||
## Advanced Few-Shot Implementation
|
||||
|
||||
### Multi-turn Conversation Pattern
|
||||
|
||||
For complex few-shot learning with full examples:
|
||||
|
||||
```json
|
||||
[
|
||||
{"role": "system", "content": "{{system_prompt}}"},
|
||||
|
||||
// Few-shot Example 1
|
||||
{"role": "user", "content": "Task: {{example_input_1}}"},
|
||||
{"role": "assistant", "content": "{{example_output_1}}"},
|
||||
|
||||
// Few-shot Example 2 (optional)
|
||||
{"role": "user", "content": "Task: {{example_input_2}}"},
|
||||
{"role": "assistant", "content": "{{example_output_2}}"},
|
||||
|
||||
// Actual test
|
||||
{"role": "user", "content": "Task: {{actual_input}}"}
|
||||
]
|
||||
```
|
||||
|
||||
**Test case configuration:**
|
||||
|
||||
```yaml
|
||||
tests:
|
||||
- vars:
|
||||
system_prompt: file://prompts/system.md
|
||||
# Few-shot examples
|
||||
example_input_1: file://data/examples/input1.txt
|
||||
example_output_1: file://data/examples/output1.txt
|
||||
example_input_2: file://data/examples/input2.txt
|
||||
example_output_2: file://data/examples/output2.txt
|
||||
# Actual test
|
||||
actual_input: file://data/test1.txt
|
||||
```
|
||||
|
||||
**Best practices:**
|
||||
- Use 1-3 few-shot examples (more may dilute effectiveness)
|
||||
- Ensure examples match the task format exactly
|
||||
- Load examples from files for better maintainability
|
||||
- Use echo provider first to verify structure
|
||||
|
||||
## Long Text Handling
|
||||
|
||||
For Chinese/long-form content evaluations (10k+ characters):
|
||||
|
||||
**Configuration:**
|
||||
|
||||
```yaml
|
||||
providers:
|
||||
- id: anthropic:messages:claude-sonnet-4-6
|
||||
config:
|
||||
max_tokens: 8192 # Increase for long outputs
|
||||
|
||||
defaultTest:
|
||||
assert:
|
||||
- type: python
|
||||
value: file://scripts/metrics.py:check_length
|
||||
```
|
||||
|
||||
**Python assertion for text metrics:**
|
||||
|
||||
```python
|
||||
import re
|
||||
|
||||
def strip_tags(text: str) -> str:
|
||||
"""Remove HTML tags for pure text."""
|
||||
return re.sub(r'<[^>]+>', '', text)
|
||||
|
||||
def check_length(output: str, context: dict) -> dict:
|
||||
"""Check output length constraints."""
|
||||
raw_input = context['vars'].get('raw_input', '')
|
||||
|
||||
input_len = len(strip_tags(raw_input))
|
||||
output_len = len(strip_tags(output))
|
||||
|
||||
reduction_ratio = 1 - (output_len / input_len) if input_len > 0 else 0
|
||||
|
||||
return {
|
||||
"pass": 0.7 <= reduction_ratio <= 0.9,
|
||||
"score": reduction_ratio,
|
||||
"reason": f"Reduction: {reduction_ratio:.1%} (target: 70-90%)",
|
||||
"named_scores": {
|
||||
"input_length": input_len,
|
||||
"output_length": output_len,
|
||||
"reduction_ratio": reduction_ratio
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Real-World Example
|
||||
|
||||
**Project:** Chinese short-video content curation from long transcripts
|
||||
|
||||
**Structure:**
|
||||
```
|
||||
tiaogaoren/
|
||||
├── promptfooconfig.yaml # Production config
|
||||
├── promptfooconfig-preview.yaml # Preview config (echo provider)
|
||||
├── prompts/
|
||||
│ ├── tiaogaoren-prompt.json # Chat format with few-shot
|
||||
│ └── v4/system-v4.md # System prompt
|
||||
├── tests/cases.yaml # 3 test samples
|
||||
├── scripts/metrics.py # Custom metrics (reduction ratio, etc.)
|
||||
├── data/ # 5 samples (2 few-shot, 3 eval)
|
||||
└── results/
|
||||
```
|
||||
|
||||
**See:** `./tiaogaoren/` (example project root) for full implementation.
|
||||
|
||||
## Resources
|
||||
|
||||
For detailed API reference and advanced patterns, see [references/promptfoo_api.md](references/promptfoo_api.md).
|
||||
@@ -0,0 +1,286 @@
|
||||
# Promptfoo API Reference
|
||||
|
||||
## Provider Configuration
|
||||
|
||||
### Echo Provider (No API Calls)
|
||||
|
||||
```yaml
|
||||
providers:
|
||||
- echo # Returns prompt as-is, no API calls
|
||||
```
|
||||
|
||||
**Use cases:**
|
||||
- Preview rendered prompts without cost
|
||||
- Debug variable substitution
|
||||
- Verify few-shot structure
|
||||
- Test configuration before production runs
|
||||
|
||||
**Cost:** Free - no tokens consumed.
|
||||
|
||||
### Anthropic
|
||||
|
||||
```yaml
|
||||
providers:
|
||||
- id: anthropic:messages:claude-sonnet-4-6
|
||||
config:
|
||||
max_tokens: 4096
|
||||
temperature: 0.7
|
||||
# For relay/proxy APIs:
|
||||
# apiBaseUrl: https://your-relay.example.com/api
|
||||
```
|
||||
|
||||
### OpenAI
|
||||
|
||||
```yaml
|
||||
providers:
|
||||
- id: openai:gpt-4.1
|
||||
config:
|
||||
temperature: 0.5
|
||||
max_tokens: 2048
|
||||
```
|
||||
|
||||
### Multiple Providers (A/B Testing)
|
||||
|
||||
```yaml
|
||||
providers:
|
||||
- id: anthropic:messages:claude-sonnet-4-6
|
||||
label: Claude
|
||||
- id: openai:gpt-4.1
|
||||
label: GPT-4.1
|
||||
```
|
||||
|
||||
## Assertion Reference
|
||||
|
||||
### Python Assertion Context
|
||||
|
||||
```python
|
||||
class AssertionContext:
|
||||
prompt: str # Raw prompt sent to LLM
|
||||
vars: dict # Test case variables
|
||||
test: dict # Complete test case
|
||||
config: dict # Assertion config
|
||||
provider: Any # Provider info
|
||||
providerResponse: Any # Full response
|
||||
```
|
||||
|
||||
### GradingResult Format
|
||||
|
||||
```python
|
||||
{
|
||||
"pass": bool, # Required: pass/fail
|
||||
"score": float, # 0.0-1.0 score
|
||||
"reason": str, # Explanation
|
||||
"named_scores": dict, # Custom metrics
|
||||
"component_results": [] # Nested results
|
||||
}
|
||||
```
|
||||
|
||||
### Assertion Types
|
||||
|
||||
| Type | Description | Parameters |
|
||||
|------|-------------|------------|
|
||||
| `contains` | Substring check | `value` |
|
||||
| `icontains` | Case-insensitive | `value` |
|
||||
| `equals` | Exact match | `value` |
|
||||
| `regex` | Pattern match | `value` |
|
||||
| `not-contains` | Absence check | `value` |
|
||||
| `starts-with` | Prefix check | `value` |
|
||||
| `contains-any` | Any substring | `value` (array) |
|
||||
| `contains-all` | All substrings | `value` (array) |
|
||||
| `cost` | Token cost | `threshold` |
|
||||
| `latency` | Response time | `threshold` (ms) |
|
||||
| `perplexity` | Model confidence | `threshold` |
|
||||
| `python` | Custom Python | `value` (file/code) |
|
||||
| `javascript` | Custom JS | `value` (code) |
|
||||
| `llm-rubric` | LLM grading | `value`, `threshold` |
|
||||
| `factuality` | Fact checking | `value` (reference) |
|
||||
| `model-graded-closedqa` | Q&A grading | `value` |
|
||||
| `similar` | Semantic similarity | `value`, `threshold` |
|
||||
|
||||
## Test Case Configuration
|
||||
|
||||
### Full Test Case Structure
|
||||
|
||||
```yaml
|
||||
- description: "Test name"
|
||||
vars:
|
||||
var1: "value"
|
||||
var2: file://path.txt
|
||||
assert:
|
||||
- type: contains
|
||||
value: "expected"
|
||||
metadata:
|
||||
category: "test-category"
|
||||
priority: high
|
||||
options:
|
||||
provider: specific-provider
|
||||
transform: "output.trim()"
|
||||
```
|
||||
|
||||
### Loading Variables from Files
|
||||
|
||||
```yaml
|
||||
vars:
|
||||
# Text file (loaded as string)
|
||||
content: file://data/input.txt
|
||||
|
||||
# JSON/YAML (parsed to object)
|
||||
config: file://config.json
|
||||
|
||||
# Python script (executed, returns value)
|
||||
dynamic: file://scripts/generate.py
|
||||
|
||||
# PDF (text extracted)
|
||||
document: file://docs/report.pdf
|
||||
|
||||
# Image (base64 encoded)
|
||||
image: file://images/photo.png
|
||||
```
|
||||
|
||||
## Advanced Patterns
|
||||
|
||||
### Dynamic Test Generation (Python)
|
||||
|
||||
```python
|
||||
# tests/generate.py
|
||||
def get_tests():
|
||||
return [
|
||||
{
|
||||
"vars": {"input": f"test {i}"},
|
||||
"assert": [{"type": "contains", "value": str(i)}]
|
||||
}
|
||||
for i in range(10)
|
||||
]
|
||||
```
|
||||
|
||||
```yaml
|
||||
tests: file://tests/generate.py:get_tests
|
||||
```
|
||||
|
||||
### Scenario-based Testing
|
||||
|
||||
```yaml
|
||||
scenarios:
|
||||
- config:
|
||||
- vars:
|
||||
language: "French"
|
||||
- vars:
|
||||
language: "Spanish"
|
||||
tests:
|
||||
- vars:
|
||||
text: "Hello"
|
||||
assert:
|
||||
- type: llm-rubric
|
||||
value: "Translation is accurate"
|
||||
```
|
||||
|
||||
### Transform Output
|
||||
|
||||
```yaml
|
||||
defaultTest:
|
||||
options:
|
||||
transform: |
|
||||
output.replace(/\n/g, ' ').trim()
|
||||
```
|
||||
|
||||
### Custom Grading Provider
|
||||
|
||||
```yaml
|
||||
defaultTest:
|
||||
options:
|
||||
provider: openai:gpt-4.1
|
||||
assert:
|
||||
- type: llm-rubric
|
||||
value: "Evaluate quality"
|
||||
provider: anthropic:claude-3-haiku # Override for this assertion
|
||||
```
|
||||
|
||||
### Relay/Proxy Provider with LLM-as-Judge
|
||||
|
||||
When using a relay or proxy API, each `llm-rubric` needs its own `provider` config — the main provider's `apiBaseUrl` is NOT inherited:
|
||||
|
||||
```yaml
|
||||
providers:
|
||||
- id: anthropic:messages:claude-sonnet-4-6
|
||||
config:
|
||||
apiBaseUrl: https://your-relay.example.com/api
|
||||
|
||||
defaultTest:
|
||||
assert:
|
||||
- type: llm-rubric
|
||||
value: "Evaluate quality"
|
||||
provider:
|
||||
id: anthropic:messages:claude-sonnet-4-6
|
||||
config:
|
||||
apiBaseUrl: https://your-relay.example.com/api # Must repeat here
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `ANTHROPIC_API_KEY` | Anthropic API key |
|
||||
| `OPENAI_API_KEY` | OpenAI API key |
|
||||
| `PROMPTFOO_PYTHON` | Python binary path |
|
||||
| `PROMPTFOO_CACHE_ENABLED` | Enable caching (default: true) |
|
||||
| `PROMPTFOO_CACHE_PATH` | Cache directory |
|
||||
|
||||
## Concurrency Control
|
||||
|
||||
**CRITICAL**: `maxConcurrency` must be placed under `commandLineOptions:` in the YAML config. Placing it at the top level is silently ignored.
|
||||
|
||||
```yaml
|
||||
# ✅ Correct — under commandLineOptions
|
||||
commandLineOptions:
|
||||
maxConcurrency: 1
|
||||
|
||||
# ❌ Wrong — top level (silently ignored, defaults to ~4)
|
||||
maxConcurrency: 1
|
||||
```
|
||||
|
||||
When using relay APIs with LLM-as-judge, set `maxConcurrency: 1` because generation and grading share the same concurrent request pool.
|
||||
|
||||
## CLI Commands
|
||||
|
||||
```bash
|
||||
# Initialize project
|
||||
npx promptfoo@latest init
|
||||
|
||||
# Run evaluation
|
||||
npx promptfoo@latest eval [options]
|
||||
|
||||
# Options:
|
||||
# --config <path> Config file path
|
||||
# --output <path> Output file path
|
||||
# --grader <provider> Override grader model
|
||||
# --no-cache Disable caching (important for re-runs)
|
||||
# --filter-metadata Filter tests by metadata
|
||||
# --repeat <n> Repeat each test n times
|
||||
# --delay <ms> Delay between requests
|
||||
# --max-concurrency Parallel requests (CLI override)
|
||||
|
||||
# View results
|
||||
npx promptfoo@latest view [options]
|
||||
|
||||
# Share results
|
||||
npx promptfoo@latest share
|
||||
|
||||
# Generate report
|
||||
npx promptfoo@latest generate dataset
|
||||
```
|
||||
|
||||
## Output Formats
|
||||
|
||||
```bash
|
||||
# JSON (default)
|
||||
--output results.json
|
||||
|
||||
# CSV
|
||||
--output results.csv
|
||||
|
||||
# HTML report
|
||||
--output results.html
|
||||
|
||||
# YAML
|
||||
--output results.yaml
|
||||
```
|
||||
@@ -0,0 +1,130 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Reusable assertion helpers for Promptfoo Python checks.
|
||||
|
||||
This module is referenced by examples in promptfoo-evaluation/SKILL.md.
|
||||
All functions return Promptfoo-compatible result dicts.
|
||||
"""
|
||||
|
||||
|
||||
def _coerce_text(output):
|
||||
"""Normalize Promptfoo output payloads into plain text."""
|
||||
if output is None:
|
||||
return ""
|
||||
if isinstance(output, str):
|
||||
return output
|
||||
if isinstance(output, dict):
|
||||
# Promptfoo often provides provider response objects.
|
||||
text = output.get("output") or output.get("content") or ""
|
||||
if isinstance(text, list):
|
||||
return "\n".join(str(x) for x in text)
|
||||
return str(text)
|
||||
return str(output)
|
||||
|
||||
|
||||
def _safe_vars(context):
|
||||
if isinstance(context, dict):
|
||||
vars_dict = context.get("vars")
|
||||
if isinstance(vars_dict, dict):
|
||||
return vars_dict
|
||||
return {}
|
||||
|
||||
|
||||
def get_assert(output, context):
|
||||
"""Default assertion function used when no function name is provided."""
|
||||
text = _coerce_text(output)
|
||||
vars_dict = _safe_vars(context)
|
||||
|
||||
expected = str(vars_dict.get("expected", "")).strip()
|
||||
if not expected:
|
||||
expected = str(vars_dict.get("expected_text", "")).strip()
|
||||
|
||||
if not expected:
|
||||
return {
|
||||
"pass": bool(text.strip()),
|
||||
"score": 1.0 if text.strip() else 0.0,
|
||||
"reason": "No expected text provided; assertion checks non-empty output.",
|
||||
"named_scores": {"non_empty": 1.0 if text.strip() else 0.0},
|
||||
}
|
||||
|
||||
matched = expected in text
|
||||
return {
|
||||
"pass": matched,
|
||||
"score": 1.0 if matched else 0.0,
|
||||
"reason": "Output contains expected text." if matched else "Expected text not found.",
|
||||
"named_scores": {"contains_expected": 1.0 if matched else 0.0},
|
||||
}
|
||||
|
||||
|
||||
def custom_assert(output, context):
|
||||
"""Alias used by SKILL.md examples."""
|
||||
return get_assert(output, context)
|
||||
|
||||
|
||||
def custom_check(output, context):
|
||||
"""Check response length against min/max word constraints."""
|
||||
text = _coerce_text(output)
|
||||
vars_dict = _safe_vars(context)
|
||||
|
||||
min_words = int(vars_dict.get("min_words", 100))
|
||||
max_words = int(vars_dict.get("max_words", 500))
|
||||
words = [w for w in text.split() if w]
|
||||
count = len(words)
|
||||
|
||||
if count == 0:
|
||||
return {
|
||||
"pass": False,
|
||||
"score": 0.0,
|
||||
"reason": "Output is empty.",
|
||||
"named_scores": {"length": 0.0},
|
||||
}
|
||||
|
||||
if min_words <= count <= max_words:
|
||||
return {
|
||||
"pass": True,
|
||||
"score": 1.0,
|
||||
"reason": "Word count within configured range.",
|
||||
"named_scores": {"length": 1.0},
|
||||
}
|
||||
|
||||
if count < min_words:
|
||||
score = max(0.0, count / float(min_words))
|
||||
return {
|
||||
"pass": False,
|
||||
"score": round(score, 3),
|
||||
"reason": "Word count below minimum.",
|
||||
"named_scores": {"length": round(score, 3)},
|
||||
}
|
||||
|
||||
overflow = max(1, count - max_words)
|
||||
score = max(0.0, 1.0 - (overflow / float(max_words)))
|
||||
return {
|
||||
"pass": False,
|
||||
"score": round(score, 3),
|
||||
"reason": "Word count above maximum.",
|
||||
"named_scores": {"length": round(score, 3)},
|
||||
}
|
||||
|
||||
|
||||
def check_length(output, context):
|
||||
"""Character-length assertion used by advanced examples."""
|
||||
text = _coerce_text(output)
|
||||
vars_dict = _safe_vars(context)
|
||||
|
||||
min_chars = int(vars_dict.get("min_chars", 1))
|
||||
max_chars = int(vars_dict.get("max_chars", 3000))
|
||||
length = len(text)
|
||||
|
||||
passed = min_chars <= length <= max_chars
|
||||
if passed:
|
||||
score = 1.0
|
||||
elif length < min_chars:
|
||||
score = max(0.0, length / float(max(1, min_chars)))
|
||||
else:
|
||||
score = max(0.0, max_chars / float(max_chars + (length - max_chars)))
|
||||
|
||||
return {
|
||||
"pass": passed,
|
||||
"score": round(score, 3),
|
||||
"reason": "Character length check.",
|
||||
"named_scores": {"char_length": round(score, 3)},
|
||||
}
|
||||
Reference in New Issue
Block a user