API reference · Schema
CreateEvaluationRequest
namestring | nullIts name
up to 200 characters
modestringrequired`label`: one label per row from `label.choices`; `grade`: a number from `grade.min` to `grade.max`; `pairwise`: the better of `pairwise.a` and `pairwise.b` (judged in both orders).
One of
labelgradepairwisedataset_file_idstringrequiredA file uploaded with purpose `evaluation` (JSONL, or CSV with a header)
ID with prefix file_
ShowHide 4 fields of
judgemodelstringrequiredModel ID (`publisher/name`)
1–256 characters
instructionsstringrequiredWhat the judge should assess, a template over the row (`{{column}}`). The texts being judged are added after it by the platform.
1–20000 characters
max_tokensintegerMost tokens per judge answer (default 512)
1–4096
temperaturenumberDefault 0
0–2
targetEvaluationCandidateInputModes `label` and `grade`
ShowHide 6 fields of
targetcolumnstringJudge the text already in this dataset column
1–128 characters
modelstringOr generate the text during the run with this model (see GET /v1/evaluations/models)
1–256 characters
promptstringWith `model`: the user message, a template over the row (`{{column}}`)
up to 20000 characters
system_promptstring | nullWith `model`: an optional system message (also a template)
up to 20000 characters
max_tokensintegerWith `model`: most tokens to generate (default 512)
1–4096
temperaturenumberWith `model`: sampling temperature, 0–2 (default 0)
0–2
pairwiseobjectMode `pairwise`: the two candidates
ShowHide 2 fields of
pairwiseaany JSONrequiredShowHide 6 fields of
acolumnstringJudge the text already in this dataset column
1–128 characters
modelstringOr generate the text during the run with this model (see GET /v1/evaluations/models)
1–256 characters
promptstringWith `model`: the user message, a template over the row (`{{column}}`)
up to 20000 characters
system_promptstring | nullWith `model`: an optional system message (also a template)
up to 20000 characters
max_tokensintegerWith `model`: most tokens to generate (default 512)
1–4096
temperaturenumberWith `model`: sampling temperature, 0–2 (default 0)
0–2
bany JSONrequiredShowHide 6 fields of
bcolumnstringJudge the text already in this dataset column
1–128 characters
modelstringOr generate the text during the run with this model (see GET /v1/evaluations/models)
1–256 characters
promptstringWith `model`: the user message, a template over the row (`{{column}}`)
up to 20000 characters
system_promptstring | nullWith `model`: an optional system message (also a template)
up to 20000 characters
max_tokensintegerWith `model`: most tokens to generate (default 512)
1–4096
temperaturenumberWith `model`: sampling temperature, 0–2 (default 0)
0–2
labelobjectMode `label`
ShowHide 2 fields of
labelchoicesarray of stringrequiredThe labels the judge chooses from (2–20); it answers with exactly one per row
2–20 items
passingarray of stringLabels that count as passing (none: no pass rate)
gradeobjectMode `grade` (whole-number bounds ask for whole-number scores)
ShowHide 3 fields of
grademinnumberrequiredLowest score the judge may give (inclusive)
maxnumberrequiredHighest score the judge may give (inclusive)
passing_atnumber | nullScores at or above this pass (none: no pass rate)
{
"mode": "label",
"dataset_file_id": "file_01jz8x3y4z5a6b7c8d9e0f1g2h",
"judge": {
"model": "qwen/qwen2.5-0.5b-instruct",
"instructions": "Classify the sentiment of this customer review."
},
"target": {
"column": "review"
},
"label": {
"choices": [
"positive",
"negative",
"neutral"
],
"passing": [
"positive"
]
}
}Used by
- Estimate an evaluation: the most it can cost, and its first row’s requests
POST /v1/evaluations/estimate - Create an evaluation over an uploaded dataset (validated and run asynchronously)
POST /v1/evaluations