Quickstarts
Your first dedicated endpoint
Serve a catalog model on replicas of your own, wait until it is serving, chat with it through the OpenAI-compatible API, and delete it.
A dedicated endpoint runs a model on replicas reserved for your project. Requests to it are not charged per token: you pay for each replica per second while it is ready, and it can stop by itself after a period without requests.
Create the endpoint
GET /v1/endpoint-models lists the catalog models an endpoint can serve and the engine each uses on CPU and on GPU hardware. You can also serve your own model from Hugging Face by giving model as {"source": "huggingface", "repository": …}.
curl -sS --fail-with-body "$API_BASE_URL/endpoint-models" -H "$AUTH" |
jq -r '"deployable: " + ([.data[].model] | join(", "))'
ENDPOINT=$(
jq -n --arg model "$MODEL" --arg sku "$SKU" --arg region "$REGION" '{
name: "quickstart-bot", model: $model, hardware: {sku: $sku}, region: $region,
min_replicas: 1, max_replicas: 1, inactivity_timeout_seconds: 600
}' |
curl -sS --fail-with-body "$API_BASE_URL/endpoints" \
-H "$AUTH" -H "Content-Type: application/json" -d @-
)
ENDPOINT_ID=$(echo "$ENDPOINT" | jq -r .id)
ENDPOINT_MODEL=$(echo "$ENDPOINT" | jq -r .model) # the endpoint's own model ID
echo "endpoint $ENDPOINT_ID serves $ENDPOINT_MODEL"The endpoint's model field is its own model ID (<org slug>/<name>). Send that ID in inference requests to reach your replicas.
Wait until it serves
Starting means placing a replica, downloading the weights and loading them, which can take minutes for a large model.
for _ in $(seq 1 600); do
STATUS=$(curl -sS --fail-with-body "$API_BASE_URL/endpoints/$ENDPOINT_ID" -H "$AUTH" |
jq -r .status)
[ "$STATUS" = serving ] && break
[ "$STATUS" = failed ] && { echo "endpoint failed"; exit 1; }
sleep 3
done
echo "status: $STATUS"Chat through it
Requests use the same inference API as shared models, with the endpoint's model ID.
jq -n --arg model "$ENDPOINT_MODEL" '{
model: $model, messages: [{role: "user", content: "Say hello in five words."}], max_completion_tokens: 32
}' |
curl -sS --fail-with-body "$INFERENCE_BASE_URL/chat/completions" \
-H "$AUTH" -H "Content-Type: application/json" -d @- |
jq -r '"reply: " + (.choices[0].message.content // "" | gsub("^\\s+|\\s+$"; ""))'Delete it
delete_endpoint() {
curl -sS --fail-with-body -X DELETE "$API_BASE_URL/endpoints/$ENDPOINT_ID" -H "$AUTH" >/dev/null
echo "deleted $ENDPOINT_ID"
}
trap delete_endpoint EXITinactivity_timeout_seconds stops an endpoint that has had no requests for that long (0 never stops it); a stopped endpoint costs nothing and can be started again.