Skip to content
Hayonix

Quickstarts

Your first batch

Upload a file of inference requests, run them as a batch at batch prices, wait for it to finish and download the results.

A batch runs many inference requests from one file, in the background, at the model's batch price. Use it for work that does not need an answer within seconds.

Write and upload the requests

Each line names a custom_id (your reference, unique in the file), the method POST, the route in url and the request body you would send to that route. Upload the file with purpose batch.

# One request per line; `custom_id` ties each result to its request.
QUESTIONS=("What is a GPU?" "What is a CPU?")
for i in "${!QUESTIONS[@]}"; do
  jq -cn --arg model "$MODEL" --arg id "question-$i" --arg q "${QUESTIONS[$i]}" '{
    custom_id: $id, method: "POST", url: "/v1/chat/completions",
    body: {model: $model, messages: [{role: "user", content: $q}], max_completion_tokens: 32}
  }'
done > "$WORK/requests.jsonl"

INPUT_ID=$(curl -sS --fail-with-body "$API_BASE_URL/files" -H "$AUTH" \
  -F purpose=batch -F "file=@$WORK/requests.jsonl" | jq -r .id)

Create the batch

endpoint is the route every line goes to: /v1/chat/completions, /v1/completions or /v1/embeddings. completion_window is 24h.

BATCH_ID=$(
  jq -n --arg file "$INPUT_ID" \
    '{input_file_id: $file, endpoint: "/v1/chat/completions", completion_window: "24h"}' |
  curl -sS --fail-with-body "$API_BASE_URL/batches" \
    -H "$AUTH" -H "Content-Type: application/json" -d @- |
  jq -r .id
)
echo "batch $BATCH_ID"

Wait for it

A batch is validating, then in_progress and finalizing, and ends completed, expired, failed or cancelled. Credit is reserved per request as it runs, so a batch that runs out of credit stops with partial results.

while :; do
  BATCH=$(curl -sS --fail-with-body "$API_BASE_URL/batches/$BATCH_ID" -H "$AUTH")
  STATUS=$(echo "$BATCH" | jq -r .status)
  echo "status: $STATUS"
  case "$STATUS" in completed|failed|expired|cancelled) break ;; esac
  sleep 5
done

Download the results

output_file_id is a JSON Lines file with one line per request that succeeded: its custom_id and the response. Requests that failed are in error_file_id. Match results to requests by custom_id.

[ "$STATUS" = completed ] || exit 1
OUTPUT_ID=$(echo "$BATCH" | jq -r .output_file_id)
curl -sS --fail-with-body "$API_BASE_URL/files/$OUTPUT_ID/content" -H "$AUTH" |
jq -r '"\(.custom_id): " + (.response.body.choices[0].message.content // "" | gsub("^\\s+|\\s+$"; ""))'

The samples delete the input and output files at the end. Output files you keep are removed after output_expires_after (30 days by default).