API reference · Schema
EvaluationList
The items
ShowHide 25 fields of
dataidstringrequiredUnique ID (`evl_…`)
ID with prefix evl_
objectstringrequiredAlways `evaluation`: the type of this object
One of
evaluationproject_idstringrequiredProject it belongs to
ID with prefix prj_
namestring | nullrequiredIts name
modestringrequired`label`: one label per row; `grade`: a score per row; `pairwise`: the better of two candidates per row
One of
labelgradepairwisestatusstringrequiredvalidating → running → finalizing → completed | stopped | failed; cancelling → cancelled. `stopped`: ended early (credit, time limit, revoked key) with every finished row kept.
One of
validatingrunningfinalizingcompletedstoppedfailedcancellingcancelledstop_reasonstring | nullrequiredWhy the evaluation ended before every row was finished: cancelled, expired, insufficient_credit, spend_limit_reached, organization_suspended, api_key_revoked, billing_<reason>; null otherwise
dataset_file_idstringrequiredID of the dataset file
ID with prefix file_
result_file_idstring | nullrequiredJSONL, one line per row (purpose `evaluation_results`), once ended
judgeobjectrequiredThe model that gives each verdict, and its settings
ShowHide 4 fields of
judgemodelstringrequiredModel ID (`publisher/name`)
instructionsstringrequiredWhat the judge assesses, a template over the row
max_tokensintegerrequiredMost tokens per judge answer
temperaturenumberrequiredThe judge's sampling temperature
ShowHide 6 fields of
targetcolumnstring | nullrequiredThe dataset column whose text is judged; null when a model generates it
modelstring | nullrequiredThe model that generates the text; null when the text comes from a column
promptstring | nullrequiredWith `model`: the user message template; null otherwise
system_promptstring | nullrequiredWith `model`: the system message template; null when none was given
max_tokensinteger | nullrequiredWith `model`: most tokens generated per row; null otherwise
temperaturenumber | nullrequiredWith `model`: sampling temperature; null otherwise
pairwiseobject | nullrequiredMode `pairwise`: the two candidates; null in other modes
ShowHide 2 fields of
pairwiseShowHide 6 fields of
acolumnstring | nullrequiredThe dataset column whose text is judged; null when a model generates it
modelstring | nullrequiredThe model that generates the text; null when the text comes from a column
promptstring | nullrequiredWith `model`: the user message template; null otherwise
system_promptstring | nullrequiredWith `model`: the system message template; null when none was given
max_tokensinteger | nullrequiredWith `model`: most tokens generated per row; null otherwise
temperaturenumber | nullrequiredWith `model`: sampling temperature; null otherwise
ShowHide 6 fields of
bcolumnstring | nullrequiredThe dataset column whose text is judged; null when a model generates it
modelstring | nullrequiredThe model that generates the text; null when the text comes from a column
promptstring | nullrequiredWith `model`: the user message template; null otherwise
system_promptstring | nullrequiredWith `model`: the system message template; null when none was given
max_tokensinteger | nullrequiredWith `model`: most tokens generated per row; null otherwise
temperaturenumber | nullrequiredWith `model`: sampling temperature; null otherwise
labelobject | nullrequiredMode `label`: its settings; null in other modes
ShowHide 2 fields of
labelchoicesarray of stringrequiredThe labels the judge chooses from
passingarray of stringrequiredLabels that count as passing; empty when no pass rate is computed
gradeobject | nullrequiredMode `grade`: its settings; null in other modes
ShowHide 3 fields of
grademinnumberrequiredLowest allowed score (inclusive)
maxnumberrequiredHighest allowed score (inclusive)
passing_atnumber | nullrequiredTime passing (ISO 8601, UTC)
progressobjectrequiredHow far the run has got
ShowHide 6 fields of
progressrows_totalintegerrequiredRows in the dataset; 0 until validation has finished
rows_judgedintegerrequiredRows that received a verdict
rows_erroredintegerrequiredRows that could not be judged
rows_skippedintegerrequiredRows never finished because the run ended early
calls_completedintegerrequiredModel calls finished so far, retries included
calls_plannedintegerrequiredRows × calls per row, before retries
Aggregate results so far (final once ended)
ShowHide 6 fields of
resultsrowsobjectrequiredRows by status
ShowHide 5 fields of
rowstotalintegerrequiredRows in the dataset
judgedintegerrequiredRows that received a verdict
erroredintegerrequiredRows that could not be judged (a failed call, a missing column, no usable judge answer)
skippedintegerrequiredNever finished: the run ended early
openintegerrequiredRows still being worked on
passedinteger | nullrequiredJudged rows that pass; null when the evaluation defines no passing labels or score, and in mode `pairwise`
pass_ratenumber | nullrequiredPassed ÷ judged, 0–1
labelsarray of object | nullrequiredMode `label`: every choice with its count, zeros included; null in other modes
ShowHide 2 fields of
labelslabelstringrequiredOne of `label.choices`
countintegerrequiredJudged rows given this label
scoresobject | nullrequiredMode `grade`: statistics of the scores; null in other modes
ShowHide 5 fields of
scoresmeannumber | nullrequiredMean score; null before any row is judged
std_devnumber | nullrequiredPopulation standard deviation of the scores; null before any row is judged
minnumber | nullrequiredLowest score given; null before any row is judged
maxnumber | nullrequiredHighest score given; null before any row is judged
histogramarray of objectrequiredScores over `grade.min`–`grade.max`: one bucket per whole number when there are at most 20, else 20 equal-width buckets
pairwiseobject | nullrequiredMode `pairwise`: how the two candidates compared; null in other modes
ShowHide 5 fields of
pairwisea_winsintegerrequiredPairs where both passes chose A
b_winsintegerrequiredPairs where both passes chose B
tiesintegerrequiredPairs where both passes called a tie
inconsistentintegerrequiredPairs whose two judging passes (A first, B first) disagreed
a_win_ratenumber | nullrequiredA wins ÷ (A wins + B wins + ties), 0–1, over consistent pairs only; null before any consistent pair
usageobjectrequiredTokens of every model call completed so far
ShowHide 3 fields of
usageinput_tokensintegerrequiredPrompt tokens over all completed calls
cached_input_tokensintegerrequiredOf `input_tokens`, those served from the prompt cache
output_tokensintegerrequiredGenerated tokens over all completed calls
coststringrequiredCharged so far (every call, at batch prices)
pattern ^-?[0-9]+$
problemsarray of objectrequiredWhy the evaluation failed
ShowHide 4 fields of
problemscodestringrequiredMachine-readable code
messagestringrequiredOne sentence for people
paramstring | nullrequiredThe request field at fault; null when not about one field
lineinteger | nullrequired1-based line of the dataset file; null when not about one line
created_atstring (date-time)requiredWhen it was created (ISO 8601, UTC)
running_atstring (date-time) | nullrequiredTime running (ISO 8601, UTC)
finalizing_atstring (date-time) | nullrequiredTime finalizing (ISO 8601, UTC)
cancelling_atstring (date-time) | nullrequiredTime cancelling (ISO 8601, UTC)
ended_atstring (date-time) | nullrequiredWhen it ended (null until then) (ISO 8601, UTC)
expires_atstring (date-time)requiredWhen a run still going stops
next_cursorstring | nullrequiredPass as `cursor` to get the next page; null when there is no more
{
"data": [
{
"id": "evl_01jc5zq4m8k3v6x9t2w7r0b1e4",
"object": "evaluation",
"project_id": "prj_01jc5zq4m8k3v6x9t2w7r0b1e4",
"name": "string",
"mode": "label",
"status": "validating",
"stop_reason": "string",
"dataset_file_id": "file_01jc5zq4m8k3v6x9t2w7r0b1e4",
"result_file_id": "string",
"judge": {
"model": "string",
"instructions": "string",
"max_tokens": 1,
"temperature": 1.5
},
"target": {
"column": "string",
"model": "string",
"prompt": "string",
"system_prompt": "string",
"max_tokens": 1,
"temperature": 1.5
},
"pairwise": {
"a": {
"column": "string",
"model": "string",
"prompt": "string",
"system_prompt": "string",
"max_tokens": 1,
"temperature": 1.5
},
"b": {
"column": "string",
"model": "string",
"prompt": "string",
"system_prompt": "string",
"max_tokens": 1,
"temperature": 1.5
}
},
"label": {
"choices": [
"string"
],
"passing": [
"string"
]
},
"grade": {
"min": 1.5,
"max": 1.5,
"passing_at": 1.5
},
"progress": {
"rows_total": 1,
"rows_judged": 1,
"rows_errored": 1,
"rows_skipped": 1,
"calls_completed": 1,
"calls_planned": 1
},
"results": {
"rows": {
"total": 1,
"judged": 1,
"errored": 1,
"skipped": 1,
"open": 1
},
"passed": 1,
"pass_rate": 1.5,
"labels": [
{
"label": "string",
"count": 1
}
],
"scores": {
"mean": 1.5,
"std_dev": 1.5,
"min": 1.5,
"max": 1.5,
"histogram": [
{
"from": null,
"to": null,
"count": null
}
]
},
"pairwise": {
"a_wins": 1,
"b_wins": 1,
"ties": 1,
"inconsistent": 1,
"a_win_rate": 1.5
}
},
"usage": {
"input_tokens": 1,
"cached_input_tokens": 1,
"output_tokens": 1
},
"cost": "1500000000",
"problems": [
{
"code": "string",
"message": "string",
"param": "string",
"line": 1
}
],
"created_at": "2026-01-01T00:00:00.000Z",
"running_at": "2026-01-01T00:00:00.000Z",
"finalizing_at": "2026-01-01T00:00:00.000Z",
"cancelling_at": "2026-01-01T00:00:00.000Z",
"ended_at": "2026-01-01T00:00:00.000Z",
"expires_at": "2026-01-01T00:00:00.000Z"
}
],
"next_cursor": "string"
}Used by
- List evaluations in the project, newest first
GET /v1/evaluations