API reference · Schema
EvaluationRowList
The items
ShowHide 13 fields of
dataidstringrequiredUnique ID (`evr_…`)
ID with prefix evr_
objectstringrequiredAlways `evaluation_row`: the type of this object
One of
evaluation_rowrowintegerrequired1-based data row
lineintegerrequired1-based line of the dataset file
statusstringrequired`open`: still being worked on; `judged`: has a verdict; `errored`: could not be judged (see `error`); `skipped`: never finished because the run ended early
One of
openjudgederroredskippedinputsobjectrequiredThe columns the run used (each cut to 2000 characters; the results file has them whole)
generatedobjectrequiredGenerated texts by candidate (`target`, `a`, `b`), cut the same way
ShowHide 4 fields of
verdictlabelstringMode `label`: the chosen label
scorenumberMode `grade`: the score given
winnerstringMode `pairwise`: `a`, `b` or `tie` when both judging passes agree; `inconsistent` when they disagree
One of
abtieinconsistentpassesobjectPairwise: the winner of each judging pass
ShowHide 2 fields of
passesa_firststringrequiredThe winner when A was shown to the judge first
One of
abtieb_firststringrequiredThe winner when B was shown to the judge first
One of
abtie
passedboolean | nullrequiredWhether the verdict passes; null until judged, when the evaluation defines no passing labels or score, and in mode `pairwise`
reasoningstring | nullrequiredThe judge's explanation (pairwise: one per judging pass); null until judged
errorobject | nullrequiredWhy the row was errored or skipped; null otherwise
ShowHide 2 fields of
errorcodestringrequiredMachine-readable code
messagestringrequiredOne sentence for people
usageobjectrequiredThis row's model calls and tokens; all 0 until the row is finished
ShowHide 3 fields of
usagecallsintegerrequiredModel calls made for this row, every try included
input_tokensintegerrequiredPrompt tokens over those calls
output_tokensintegerrequiredGenerated tokens over those calls
coststringrequiredInteger nano-USD as a decimal string
pattern ^-?[0-9]+$
next_cursorstring | nullrequiredPass as `cursor` to get the next page; null when there is no more
{
"data": [
{
"id": "evr_01jc5zq4m8k3v6x9t2w7r0b1e4",
"object": "evaluation_row",
"row": 1,
"line": 1,
"status": "open",
"inputs": {},
"generated": {},
"verdict": {
"label": "string",
"score": 1.5,
"winner": "a",
"passes": {
"a_first": "a",
"b_first": "a"
}
},
"passed": true,
"reasoning": "string",
"error": {
"code": "string",
"message": "string"
},
"usage": {
"calls": 1,
"input_tokens": 1,
"output_tokens": 1
},
"cost": "1500000000"
}
],
"next_cursor": "string"
}Used by
- List an evaluation’s rows in dataset order: verdict, reasoning, texts, cost
GET /v1/evaluations/{evaluation_id}/rows