Quickstarts
Evaluate outputs with a judge model
Upload a dataset, have a judge model label, grade or compare every row, read the aggregate results and the rows that did not pass, and download the results file.
An evaluation runs a judge model over every row of a dataset file. It works in one of three modes:
| Mode | The judge gives each row | Results |
|---|---|---|
label | one label from label.choices (rows whose label is in label.passing pass) | how many rows got each label, pass rate |
grade | a number from grade.min to grade.max (rows at or above grade.passing_at pass) | mean, standard deviation, min, max, histogram, pass rate |
pairwise | the better of pairwise.a and pairwise.b, or a tie | A wins, B wins, ties, inconsistent pairs |
What is judged — target for label and grade, pairwise.a and pairwise.b for pairwise — is either a column of the dataset ({"column": "answer"}) or text a model generates during the run from a prompt template ({"model": "…", "prompt": "Answer: {{question}}", "max_tokens": 256}). GET /v1/evaluations/models lists the models you can use as a judge or to generate.
Upload a dataset
A dataset is a JSON Lines file (one object per row; its keys are the columns) or a CSV file with a header row, uploaded with purpose evaluation. It is validated asynchronously: the file's validation report lists its rows and columns, and every problem with its line number. A file with problems cannot be evaluated.
# One JSON object per line; its keys are the columns templates can use.
for review in "Absolutely wonderful, I loved every minute." \
"Broken on arrival and support never answered." "It arrived on Tuesday."; do
jq -cn --arg review "$review" '{review: $review}'
done > "$WORK/reviews.jsonl"
DATASET_ID=$(curl -sS --fail-with-body "$API_BASE_URL/files" -H "$AUTH" \
-F purpose=evaluation -F "file=@$WORK/reviews.jsonl" | jq -r .id)
until curl -sS --fail-with-body "$API_BASE_URL/files/$DATASET_ID" -H "$AUTH" |
jq -e '.validation.state | IN("valid", "invalid")' >/dev/null; do sleep 1; doneEstimate the cost
The estimate takes the same settings as creating an evaluation and checks them the same way: a {{column}} placeholder that names a column the dataset does not have is an error that names the column, before anything is spent. It returns the most the run can cost and the first row's requests exactly as they will be sent.
jq -n --arg file "$DATASET_ID" --arg model "$MODEL" '{
mode: "label", dataset_file_id: $file,
judge: {model: $model, instructions: "Classify the sentiment of this customer review.", max_tokens: 200},
target: {column: "review"},
label: {choices: ["positive", "negative", "neutral"], passing: ["positive"]}
}' > "$WORK/settings.json"
curl -sS --fail-with-body "$API_BASE_URL/evaluations/estimate" -H "$AUTH" \
-H "Content-Type: application/json" -d @"$WORK/settings.json" |
jq -r '"at most \(.max_cost) nano-USD for \(.planned_calls) calls"'Every model call is an ordinary inference request charged at the model's batch price as it completes, and runs at low priority behind live traffic; nothing is reserved up front. The estimate is an upper bound: input tokens are counted as at most the UTF-8 bytes of each request, output tokens as max_tokens. A judge answer that cannot be read is asked again (at most 3 calls per judging step); those retries are charged like any call and are not in the estimate.
Create it and wait
An evaluation is validating, then running and finalizing, and ends completed, stopped (it ran out of credit or time, and kept every row it finished), failed (the dataset could not be evaluated, or more than half of the rows errored) or cancelled.
EVALUATION_ID=$(curl -sS --fail-with-body "$API_BASE_URL/evaluations" -H "$AUTH" \
-H "Content-Type: application/json" -d @"$WORK/settings.json" | jq -r .id)
while :; do
EVALUATION=$(curl -sS --fail-with-body "$API_BASE_URL/evaluations/$EVALUATION_ID" -H "$AUTH")
STATUS=$(echo "$EVALUATION" | jq -r .status)
echo "status: $STATUS"
case "$STATUS" in completed|stopped|failed|cancelled) break ;; esac
sleep 3
doneIn pairwise mode each pair is judged twice, once with each candidate shown first, and a win counts only when both passes agree; pairs where they disagree are counted as inconsistent, so the judge's position bias does not pass for a result.
Read the results
results holds the aggregate numbers (while the run is going, too). GET /v1/evaluations/{evaluation_id}/rows lists the rows in dataset order with the verdict, the judge's reasoning, the texts and the cost; filter narrows it to passed, failed, errored (rows with an error, and rows never run) or inconsistent.
echo "$EVALUATION" | jq -r '"pass rate: \(.results.pass_rate * 100)%"'
curl -sS --fail-with-body "$API_BASE_URL/evaluations/$EVALUATION_ID/rows?filter=failed" -H "$AUTH" |
jq -r '.data[] | "row \(.row): \(.verdict.label) (\(.reasoning))"'The judge's answer is constrained to a JSON schema and parsed strictly. A row whose answer still cannot be read after the last try is an error row with the reason; a verdict is never guessed.
Download the results file
result_file_id is a JSON Lines file (purpose evaluation_results) with one line per row: the inputs the run used, the generated texts, the verdict and reasoning, every judge answer, tokens, cost and any error. It is kept for 30 days.
curl -sS --fail-with-body "$API_BASE_URL/files/$RESULT_ID/content" -H "$AUTH" |
jq -r '"row \(.row): \(.status) \(.verdict.label // "")"'