Skip to content
Hayonix

Quickstarts

Your first dedicated endpoint

Serve a catalog model on replicas of your own, wait until it is serving, chat with it through the OpenAI-compatible API, and delete it.

A dedicated endpoint runs a model on replicas reserved for your project. Requests to it are not charged per token: you pay for each replica per second while it is ready, and it can stop by itself after a period without requests.

Create the endpoint

GET /v1/endpoint-models lists the catalog models an endpoint can serve and the engine each uses on CPU and on GPU hardware. You can also serve your own model from Hugging Face by giving model as {"source": "huggingface", "repository": …}.

curl -sS --fail-with-body "$API_BASE_URL/endpoint-models" -H "$AUTH" |
jq -r '"deployable: " + ([.data[].model] | join(", "))'

ENDPOINT=$(
  jq -n --arg model "$MODEL" --arg sku "$SKU" --arg region "$REGION" '{
    name: "quickstart-bot", model: $model, hardware: {sku: $sku}, region: $region,
    min_replicas: 1, max_replicas: 1, inactivity_timeout_seconds: 600
  }' |
  curl -sS --fail-with-body "$API_BASE_URL/endpoints" \
    -H "$AUTH" -H "Content-Type: application/json" -d @-
)
ENDPOINT_ID=$(echo "$ENDPOINT" | jq -r .id)
ENDPOINT_MODEL=$(echo "$ENDPOINT" | jq -r .model) # the endpoint's own model ID
echo "endpoint $ENDPOINT_ID serves $ENDPOINT_MODEL"

The endpoint's model field is its own model ID (<org slug>/<name>). Send that ID in inference requests to reach your replicas.

Wait until it serves

Starting means placing a replica, downloading the weights and loading them, which can take minutes for a large model.

for _ in $(seq 1 600); do
  STATUS=$(curl -sS --fail-with-body "$API_BASE_URL/endpoints/$ENDPOINT_ID" -H "$AUTH" |
    jq -r .status)
  [ "$STATUS" = serving ] && break
  [ "$STATUS" = failed ] && { echo "endpoint failed"; exit 1; }
  sleep 3
done
echo "status: $STATUS"

Chat through it

Requests use the same inference API as shared models, with the endpoint's model ID.

jq -n --arg model "$ENDPOINT_MODEL" '{
  model: $model, messages: [{role: "user", content: "Say hello in five words."}], max_completion_tokens: 32
}' |
curl -sS --fail-with-body "$INFERENCE_BASE_URL/chat/completions" \
  -H "$AUTH" -H "Content-Type: application/json" -d @- |
jq -r '"reply: " + (.choices[0].message.content // "" | gsub("^\\s+|\\s+$"; ""))'

Delete it

delete_endpoint() {
  curl -sS --fail-with-body -X DELETE "$API_BASE_URL/endpoints/$ENDPOINT_ID" -H "$AUTH" >/dev/null
  echo "deleted $ENDPOINT_ID"
}
trap delete_endpoint EXIT

inactivity_timeout_seconds stops an endpoint that has had no requests for that long (0 never stops it); a stopped endpoint costs nothing and can be started again.