Skip to content
Hayonix

API reference · Evaluations

Estimate an evaluation: the most it can cost, and its first row’s requests

POST/v1/evaluations/estimate
API keyevaluation:readProject
TypeScript
client.evaluations.estimate()
Python
client.evaluations.estimate()
CLI
hayonix evaluations estimate

Takes the same body as creating an evaluation and checks it the same way (the dataset must be validated). Nothing is created or charged.

Project API key (Authorization: Bearer sk_…), or a console session. Requires the evaluation:read permission (a key's scope, or a role that includes it).

Examples

The samples read API_KEY and API_BASE_URL from the environment (see Setup). Values are examples.

curl -sS "$API_BASE_URL/evaluations/estimate" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "mode": "label",
  "dataset_file_id": "file_01jz8x3y4z5a6b7c8d9e0f1g2h",
  "judge": {
    "model": "qwen/qwen2.5-0.5b-instruct",
    "instructions": "Classify the sentiment of this customer review."
  },
  "target": {
    "column": "review"
  },
  "label": {
    "choices": [
      "positive",
      "negative",
      "neutral"
    ],
    "passing": [
      "positive"
    ]
  }
}'

Request

Headers
HeaderRequiredDescription
AuthorizationyesBearer and your API key. The key implies its project.
X-Project-Idsession onlyThe project to act in, for console sessions. Requests with an API key leave it out.

Body

JSON (application/json): CreateEvaluationRequest.

  • namestring | null

    Its name

    up to 200 characters

  • modestringrequired

    `label`: one label per row from `label.choices`; `grade`: a number from `grade.min` to `grade.max`; `pairwise`: the better of `pairwise.a` and `pairwise.b` (judged in both orders).

    One oflabelgradepairwise

  • dataset_file_idstringrequired

    A file uploaded with purpose `evaluation` (JSONL, or CSV with a header)

    ID with prefix file_

  • Show 4 fields of judge
    • modelstringrequired

      Model ID (`publisher/name`)

      1–256 characters

    • instructionsstringrequired

      What the judge should assess, a template over the row (`{{column}}`). The texts being judged are added after it by the platform.

      1–20000 characters

    • max_tokensinteger

      Most tokens per judge answer (default 512)

      1–4096

    • temperaturenumber

      Default 0

      0–2

  • Modes `label` and `grade`

    Show 6 fields of target
    • columnstring

      Judge the text already in this dataset column

      1–128 characters

    • modelstring

      Or generate the text during the run with this model (see GET /v1/evaluations/models)

      1–256 characters

    • promptstring

      With `model`: the user message, a template over the row (`{{column}}`)

      up to 20000 characters

    • system_promptstring | null

      With `model`: an optional system message (also a template)

      up to 20000 characters

    • max_tokensinteger

      With `model`: most tokens to generate (default 512)

      1–4096

    • temperaturenumber

      With `model`: sampling temperature, 0–2 (default 0)

      0–2

  • pairwiseobject

    Mode `pairwise`: the two candidates

    Show 2 fields of pairwise
    • aany JSONrequired
      Show 6 fields of a
      • columnstring

        Judge the text already in this dataset column

        1–128 characters

      • modelstring

        Or generate the text during the run with this model (see GET /v1/evaluations/models)

        1–256 characters

      • promptstring

        With `model`: the user message, a template over the row (`{{column}}`)

        up to 20000 characters

      • system_promptstring | null

        With `model`: an optional system message (also a template)

        up to 20000 characters

      • max_tokensinteger

        With `model`: most tokens to generate (default 512)

        1–4096

      • temperaturenumber

        With `model`: sampling temperature, 0–2 (default 0)

        0–2

    • bany JSONrequired
      Show 6 fields of b
      • columnstring

        Judge the text already in this dataset column

        1–128 characters

      • modelstring

        Or generate the text during the run with this model (see GET /v1/evaluations/models)

        1–256 characters

      • promptstring

        With `model`: the user message, a template over the row (`{{column}}`)

        up to 20000 characters

      • system_promptstring | null

        With `model`: an optional system message (also a template)

        up to 20000 characters

      • max_tokensinteger

        With `model`: most tokens to generate (default 512)

        1–4096

      • temperaturenumber

        With `model`: sampling temperature, 0–2 (default 0)

        0–2

  • labelobject

    Mode `label`

    Show 2 fields of label
    • choicesarray of stringrequired

      The labels the judge chooses from (2–20); it answers with exactly one per row

      2–20 items

    • passingarray of string

      Labels that count as passing (none: no pass rate)

  • gradeobject

    Mode `grade` (whole-number bounds ask for whole-number scores)

    Show 3 fields of grade
    • minnumberrequired

      Lowest score the judge may give (inclusive)

    • maxnumberrequired

      Highest score the judge may give (inclusive)

    • passing_atnumber | null

      Scores at or above this pass (none: no pass rate)

Response

200 EvaluationEstimate — The estimate

  • objectstringrequired

    Always `evaluation_estimate`: the type of this object

    One ofevaluation_estimate

  • rowsintegerrequired

    Rows in the dataset

  • calls_per_rowintegerrequired

    Model calls per row: one per generated candidate, plus one judge call (two in mode `pairwise`)

  • planned_callsintegerrequired

    Rows × calls per row, before judge retries

  • max_coststringrequired

    The most the run can cost, without judge retries

    pattern ^-?[0-9]+$

  • price_classstringrequired

    Every call is charged at the models' batch prices, as it completes

    One ofbatch

  • boundstringrequired

    How the token bound is stated

  • linesarray of objectrequired

    The bound per step of the run

    Show 7 fields of lines
    • stepstringrequired

      `target`, `a`, `b`: generating that candidate; `judge`: judging; `judge_ab` / `judge_ba`: the pairwise judging pass with A or B shown first

      One oftargetabjudgejudge_abjudge_ba

    • modelstringrequired

      Model ID (`publisher/name`)

    • billingstringrequired

      `per_token`: charged per token at batch prices; `replica_time`: a dedicated endpoint, paid by its replica time and not charged per call

      One ofper_tokenreplica_time

    • callsintegerrequired

      Calls of this step (one per row)

    • max_input_tokensintegerrequired

      Upper bound on prompt tokens over all calls of this step

    • max_output_tokensintegerrequired

      Upper bound on generated tokens over all calls of this step

    • max_coststringrequired

      Integer nano-USD as a decimal string

      pattern ^-?[0-9]+$

  • columnsarray of stringrequired

    The dataset's columns

  • columns_usedarray of stringrequired

    The columns the settings read (templates and candidate columns)

  • warningsarray of stringrequired

    Problems that will not stop the run, such as rows missing a column (those rows will be errors)

  • previewobject | nullrequired

    The first row's requests, rendered exactly as the run sends them

    Show 3 fields of preview
    • rowintegerrequired

      1-based data row shown

    • lineintegerrequired

      1-based line of the dataset file

    • requestsarray of objectrequired

      One request per step, in the order the run sends them

      Show 6 fields of requests
      • stepstringrequired

        The step the request belongs to, as in `lines`

      • modelstringrequired

        Model ID (`publisher/name`)

      • messagesarray of objectrequired

        The chat messages, with every template filled in

        Show 2 fields of messages
        • rolestringrequired

          Who the message is from

          One ofsystemuser

        • contentstringrequired

          The message text

      • max_tokensintegerrequired

        Most tokens the call may generate

      • temperaturenumberrequired

        Sampling temperature of the call

      • constrainedbooleanrequired

        Answer constrained by a JSON schema

Example response
Example response
{
  "object": "evaluation_estimate",
  "rows": 1,
  "calls_per_row": 1,
  "planned_calls": 1,
  "max_cost": "1500000000",
  "price_class": "batch",
  "bound": "string",
  "lines": [
    {
      "step": "target",
      "model": "string",
      "billing": "per_token",
      "calls": 1,
      "max_input_tokens": 1,
      "max_output_tokens": 1,
      "max_cost": "1500000000"
    }
  ],
  "columns": [
    "string"
  ],
  "columns_used": [
    "string"
  ],
  "warnings": [
    "string"
  ],
  "preview": {
    "row": 1,
    "line": 1,
    "requests": [
      {
        "step": "string",
        "model": "string",
        "messages": [
          {
            "role": "system",
            "content": "string"
          }
        ],
        "max_tokens": 1,
        "temperature": 1.5,
        "constrained": true
      }
    ]
  }
}

Errors

Errors use the standard envelope. What this operation can answer:

Errors
StatusTypeWhen
400invalid_request_errorA parameter or body field is missing or not allowed; param names it.
401authentication_errorThe key or session is missing, revoked or expired.
403permission_errorThe credential lacks the permission this needs, or the organization is suspended.
429rate_limit_error rate_limitedToo many requests; retry after retry-after seconds.