API reference · Evaluations
Estimate an evaluation: the most it can cost, and its first row’s requests
/v1/evaluations/estimate- TypeScript
client.evaluations.estimate()- Python
client.evaluations.estimate()- CLI
hayonix evaluations estimate
Takes the same body as creating an evaluation and checks it the same way (the dataset must be validated). Nothing is created or charged.
Project API key (Authorization: Bearer sk_…), or a console session. Requires the evaluation:read permission (a key's scope, or a role that includes it).
Examples
The samples read API_KEY and API_BASE_URL from the environment (see Setup). Values are examples.
curl -sS "$API_BASE_URL/evaluations/estimate" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"mode": "label",
"dataset_file_id": "file_01jz8x3y4z5a6b7c8d9e0f1g2h",
"judge": {
"model": "qwen/qwen2.5-0.5b-instruct",
"instructions": "Classify the sentiment of this customer review."
},
"target": {
"column": "review"
},
"label": {
"choices": [
"positive",
"negative",
"neutral"
],
"passing": [
"positive"
]
}
}'Request
| Header | Required | Description |
|---|---|---|
Authorization | yes | Bearer and your API key. The key implies its project. |
X-Project-Id | session only | The project to act in, for console sessions. Requests with an API key leave it out. |
Body
JSON (application/json): CreateEvaluationRequest.
namestring | nullIts name
up to 200 characters
modestringrequired`label`: one label per row from `label.choices`; `grade`: a number from `grade.min` to `grade.max`; `pairwise`: the better of `pairwise.a` and `pairwise.b` (judged in both orders).
One of
labelgradepairwisedataset_file_idstringrequiredA file uploaded with purpose `evaluation` (JSONL, or CSV with a header)
ID with prefix file_
ShowHide 4 fields of
judgemodelstringrequiredModel ID (`publisher/name`)
1–256 characters
instructionsstringrequiredWhat the judge should assess, a template over the row (`{{column}}`). The texts being judged are added after it by the platform.
1–20000 characters
max_tokensintegerMost tokens per judge answer (default 512)
1–4096
temperaturenumberDefault 0
0–2
targetEvaluationCandidateInputModes `label` and `grade`
ShowHide 6 fields of
targetcolumnstringJudge the text already in this dataset column
1–128 characters
modelstringOr generate the text during the run with this model (see GET /v1/evaluations/models)
1–256 characters
promptstringWith `model`: the user message, a template over the row (`{{column}}`)
up to 20000 characters
system_promptstring | nullWith `model`: an optional system message (also a template)
up to 20000 characters
max_tokensintegerWith `model`: most tokens to generate (default 512)
1–4096
temperaturenumberWith `model`: sampling temperature, 0–2 (default 0)
0–2
pairwiseobjectMode `pairwise`: the two candidates
ShowHide 2 fields of
pairwiseaany JSONrequiredShowHide 6 fields of
acolumnstringJudge the text already in this dataset column
1–128 characters
modelstringOr generate the text during the run with this model (see GET /v1/evaluations/models)
1–256 characters
promptstringWith `model`: the user message, a template over the row (`{{column}}`)
up to 20000 characters
system_promptstring | nullWith `model`: an optional system message (also a template)
up to 20000 characters
max_tokensintegerWith `model`: most tokens to generate (default 512)
1–4096
temperaturenumberWith `model`: sampling temperature, 0–2 (default 0)
0–2
bany JSONrequiredShowHide 6 fields of
bcolumnstringJudge the text already in this dataset column
1–128 characters
modelstringOr generate the text during the run with this model (see GET /v1/evaluations/models)
1–256 characters
promptstringWith `model`: the user message, a template over the row (`{{column}}`)
up to 20000 characters
system_promptstring | nullWith `model`: an optional system message (also a template)
up to 20000 characters
max_tokensintegerWith `model`: most tokens to generate (default 512)
1–4096
temperaturenumberWith `model`: sampling temperature, 0–2 (default 0)
0–2
labelobjectMode `label`
ShowHide 2 fields of
labelchoicesarray of stringrequiredThe labels the judge chooses from (2–20); it answers with exactly one per row
2–20 items
passingarray of stringLabels that count as passing (none: no pass rate)
gradeobjectMode `grade` (whole-number bounds ask for whole-number scores)
ShowHide 3 fields of
grademinnumberrequiredLowest score the judge may give (inclusive)
maxnumberrequiredHighest score the judge may give (inclusive)
passing_atnumber | nullScores at or above this pass (none: no pass rate)
Response
200 EvaluationEstimate — The estimate
objectstringrequiredAlways `evaluation_estimate`: the type of this object
One of
evaluation_estimaterowsintegerrequiredRows in the dataset
calls_per_rowintegerrequiredModel calls per row: one per generated candidate, plus one judge call (two in mode `pairwise`)
planned_callsintegerrequiredRows × calls per row, before judge retries
max_coststringrequiredThe most the run can cost, without judge retries
pattern ^-?[0-9]+$
price_classstringrequiredEvery call is charged at the models' batch prices, as it completes
One of
batchboundstringrequiredHow the token bound is stated
linesarray of objectrequiredThe bound per step of the run
ShowHide 7 fields of
linesstepstringrequired`target`, `a`, `b`: generating that candidate; `judge`: judging; `judge_ab` / `judge_ba`: the pairwise judging pass with A or B shown first
One of
targetabjudgejudge_abjudge_bamodelstringrequiredModel ID (`publisher/name`)
billingstringrequired`per_token`: charged per token at batch prices; `replica_time`: a dedicated endpoint, paid by its replica time and not charged per call
One of
per_tokenreplica_timecallsintegerrequiredCalls of this step (one per row)
max_input_tokensintegerrequiredUpper bound on prompt tokens over all calls of this step
max_output_tokensintegerrequiredUpper bound on generated tokens over all calls of this step
max_coststringrequiredInteger nano-USD as a decimal string
pattern ^-?[0-9]+$
columnsarray of stringrequiredThe dataset's columns
columns_usedarray of stringrequiredThe columns the settings read (templates and candidate columns)
warningsarray of stringrequiredProblems that will not stop the run, such as rows missing a column (those rows will be errors)
previewobject | nullrequiredThe first row's requests, rendered exactly as the run sends them
ShowHide 3 fields of
previewrowintegerrequired1-based data row shown
lineintegerrequired1-based line of the dataset file
requestsarray of objectrequiredOne request per step, in the order the run sends them
ShowHide 6 fields of
requestsstepstringrequiredThe step the request belongs to, as in `lines`
modelstringrequiredModel ID (`publisher/name`)
messagesarray of objectrequiredThe chat messages, with every template filled in
ShowHide 2 fields of
messagesrolestringrequiredWho the message is from
One of
systemusercontentstringrequiredThe message text
max_tokensintegerrequiredMost tokens the call may generate
temperaturenumberrequiredSampling temperature of the call
constrainedbooleanrequiredAnswer constrained by a JSON schema
Example response
{
"object": "evaluation_estimate",
"rows": 1,
"calls_per_row": 1,
"planned_calls": 1,
"max_cost": "1500000000",
"price_class": "batch",
"bound": "string",
"lines": [
{
"step": "target",
"model": "string",
"billing": "per_token",
"calls": 1,
"max_input_tokens": 1,
"max_output_tokens": 1,
"max_cost": "1500000000"
}
],
"columns": [
"string"
],
"columns_used": [
"string"
],
"warnings": [
"string"
],
"preview": {
"row": 1,
"line": 1,
"requests": [
{
"step": "string",
"model": "string",
"messages": [
{
"role": "system",
"content": "string"
}
],
"max_tokens": 1,
"temperature": 1.5,
"constrained": true
}
]
}
}Errors
Errors use the standard envelope. What this operation can answer:
| Status | Type | When |
|---|---|---|
400 | invalid_request_error | A parameter or body field is missing or not allowed; param names it. |
401 | authentication_error | The key or session is missing, revoked or expired. |
403 | permission_error | The credential lacks the permission this needs, or the organization is suspended. |
429 | rate_limit_error rate_limited | Too many requests; retry after retry-after seconds. |