API reference · Schema
EvaluationEstimate
objectstringrequiredAlways `evaluation_estimate`: the type of this object
One of
evaluation_estimaterowsintegerrequiredRows in the dataset
calls_per_rowintegerrequiredModel calls per row: one per generated candidate, plus one judge call (two in mode `pairwise`)
planned_callsintegerrequiredRows × calls per row, before judge retries
max_coststringrequiredThe most the run can cost, without judge retries
pattern ^-?[0-9]+$
price_classstringrequiredEvery call is charged at the models' batch prices, as it completes
One of
batchboundstringrequiredHow the token bound is stated
linesarray of objectrequiredThe bound per step of the run
ShowHide 7 fields of
linesstepstringrequired`target`, `a`, `b`: generating that candidate; `judge`: judging; `judge_ab` / `judge_ba`: the pairwise judging pass with A or B shown first
One of
targetabjudgejudge_abjudge_bamodelstringrequiredModel ID (`publisher/name`)
billingstringrequired`per_token`: charged per token at batch prices; `replica_time`: a dedicated endpoint, paid by its replica time and not charged per call
One of
per_tokenreplica_timecallsintegerrequiredCalls of this step (one per row)
max_input_tokensintegerrequiredUpper bound on prompt tokens over all calls of this step
max_output_tokensintegerrequiredUpper bound on generated tokens over all calls of this step
max_coststringrequiredInteger nano-USD as a decimal string
pattern ^-?[0-9]+$
columnsarray of stringrequiredThe dataset's columns
columns_usedarray of stringrequiredThe columns the settings read (templates and candidate columns)
warningsarray of stringrequiredProblems that will not stop the run, such as rows missing a column (those rows will be errors)
previewobject | nullrequiredThe first row's requests, rendered exactly as the run sends them
ShowHide 3 fields of
previewrowintegerrequired1-based data row shown
lineintegerrequired1-based line of the dataset file
requestsarray of objectrequiredOne request per step, in the order the run sends them
ShowHide 6 fields of
requestsstepstringrequiredThe step the request belongs to, as in `lines`
modelstringrequiredModel ID (`publisher/name`)
messagesarray of objectrequiredThe chat messages, with every template filled in
ShowHide 2 fields of
messagesrolestringrequiredWho the message is from
One of
systemusercontentstringrequiredThe message text
max_tokensintegerrequiredMost tokens the call may generate
temperaturenumberrequiredSampling temperature of the call
constrainedbooleanrequiredAnswer constrained by a JSON schema
{
"object": "evaluation_estimate",
"rows": 1,
"calls_per_row": 1,
"planned_calls": 1,
"max_cost": "1500000000",
"price_class": "batch",
"bound": "string",
"lines": [
{
"step": "target",
"model": "string",
"billing": "per_token",
"calls": 1,
"max_input_tokens": 1,
"max_output_tokens": 1,
"max_cost": "1500000000"
}
],
"columns": [
"string"
],
"columns_used": [
"string"
],
"warnings": [
"string"
],
"preview": {
"row": 1,
"line": 1,
"requests": [
{
"step": "string",
"model": "string",
"messages": [
{
"role": "system",
"content": "string"
}
],
"max_tokens": 1,
"temperature": 1.5,
"constrained": true
}
]
}
}Used by
- Estimate an evaluation: the most it can cost, and its first row’s requests
POST /v1/evaluations/estimate