Skip to content
Hayonix

Quickstarts

Serve HTTP from a serverless endpoint

Run an image that serves HTTP itself behind a stable URL, scale it with requests in flight down to zero, roll releases without failed requests, and keep idle workers paused so the next request skips the cold start.

A serverless endpoint is one of two types, chosen when you create it:

TypeCallersYour imageFits
queuesubmit jobs and fetch the results, or wait for themruns a handler on the worker runtimebatch work, long jobs, retries, webhooks
httpsend HTTP requests to the endpoint's URLserves HTTP itself, unchangedexisting servers (an OpenAI-compatible model server, say), streaming, WebSockets, low latency

This guide is about http endpoints. Your image needs only to listen on a port and answer a health path with a 2xx or 3xx once it is ready; no platform runtime is involved.

Create one

http.port is where your server listens inside the container; http.health_path is probed with GET before a worker takes requests, and keeps being probed: a worker that fails it is taken out of rotation, and replaced if it does not recover within 30 s. concurrency is the most requests one worker serves at once. The response carries the endpoint's url.

ENDPOINT=$(
  jq -n --arg image "$IMAGE" --arg sku "$SKU" --arg region "$REGION" '{
    name: "http-server",
    mode: "http",
    image: $image,
    http: { port: 8000, health_path: "/health" },
    skus: [$sku],
    region: $region,
    concurrency: 8,
    max_workers: 2,
    cold_start_wait_seconds: 120
  }' |
  curl -sS --fail-with-body "$API_BASE_URL/serverless/endpoints" \
    -H "$AUTH" -H "Content-Type: application/json" -d @-
)
ENDPOINT_ID=$(echo "$ENDPOINT" | jq -r .id)
URL=$(echo "$ENDPOINT" | jq -r .url)
echo "endpoint $ENDPOINT_ID at $URL"

Call it

Send any method and path to the URL with an API key of the endpoint's project that holds the Serverless endpoints invoke permission (serverless_endpoint:invoke) as Authorization: Bearer. Without a key the answer is 401; a key of another project gets 404, as if the endpoint did not exist.

# Any method and path reaches your server. With no worker running, this waits for one to
# start (up to cold_start_wait_seconds), then answers.
curl -sS --fail-with-body "${URL}" -H "$AUTH" | jq -r '"answered by \(.worker_id)"'
# Streaming responses arrive as they are written (-N turns off curl's buffering).
curl -sSN --fail-with-body "${URL}stream?n=3&delay=0.2" -H "$AUTH" |
  sed -n 's/^data: \(.*\)/\1/p' | jq -r '"event \(.i)"'

What your container receives:

  • the request as sent, streamed in both directions: server-sent events, chunked responses and WebSocket upgrades pass through as they are written. Request bodies are limited to 100 MiB.
  • never the platform API key or console cookies. Instead: x-endpoint-caller-key-id (the ID of the key that called, to tell callers apart), x-endpoint-request-id (also returned to the caller), x-endpoint-id and x-endpoint-worker-id, and X-Forwarded-For/-Proto/-Host. Headers starting with x-endpoint- that a caller sends are dropped.
  • the environment variables WORKER_RUNTIME_PORT, WORKER_RUNTIME_WORKER_ID, WORKER_RUNTIME_ENDPOINT_ID and WORKER_RUNTIME_RELEASE_ID, besides your own.

Scaling and cold starts

Workers are added and removed to keep about scaling.target requests in flight per worker (requests waiting for a worker count too), between min_workers and max_workers. A request goes to the worker with the fewest requests in flight, never to one already at its concurrency.

With no worker ready — the endpoint scaled to zero, or every worker busy — a request waits for one for up to cold_start_wait_seconds (default 60, at most 300) while the autoscaler starts one. If none is ready in time it is answered 503 with code no_worker_ready (or workers_busy) and a Retry-After header: retry it. When the organization cannot start a worker at all (out of credit, suspended, at a spend limit), the answer is the matching error at once — 402 for no credit — not a timeout.

A worker leaves rotation before it is removed (scaling down, or replaced by a new release) and gets drain_timeout_seconds to finish the requests it has in flight. A response must start within execution_timeout_seconds; after that it is answered 504. Long streams and WebSockets may then run as long as data keeps moving.

Releases

Changing the image, environment, hardware, concurrency or http settings makes a new release. New workers start on it and must pass their health check before workers of the old release leave rotation, so a release rolls out without a failed request. Rolling back makes a new release with an earlier spec.

Fast resume

An idle worker is normally removed after idle_timeout_seconds, and the next request pays for a cold start (starting the container and, typically, loading a model). With fast_resume.max_paused_workers above 0, up to that many idle workers are paused instead: the container is frozen in place with its memory — and its loaded model, on a GPU its VRAM — intact, and the next request (or job, on a queue endpoint) resumes it in a fraction of a cold start. A paused worker is removed after fast_resume.paused_keep_seconds.

# Keep one idle worker paused instead of removing it: the next request resumes it in a
# fraction of a cold start. Paused time is not charged.
curl -sS --fail-with-body -X PATCH "$API_BASE_URL/serverless/endpoints/$ENDPOINT_ID" \
  -H "$AUTH" -H "Content-Type: application/json" \
  -d '{"fast_resume": {"max_paused_workers": 1, "paused_keep_seconds": 600}}' |
  jq -r '"fast resume: \(.fast_resume.max_paused_workers) paused worker kept"'

What it costs: nothing while paused. Billing is per second of running time, and paused seconds are not counted (a resumed worker bills again from the moment it is unfrozen). In exchange, the platform may remove a paused worker at any time when its host needs the capacity; the next request then starts a new worker, as without fast resume. Paused workers do not count against your worker quota. Fast resume works the same way on queue endpoints.

The endpoint's metrics show cold starts and resumes separately (GET /v1/serverless/endpoints/{endpoint_id}/metrics), and GET …/request-metrics has the request rate, status classes, latency percentiles, requests in flight and how often a request waited for a worker, per endpoint and per worker.

Delete it

delete_endpoint() {
  curl -sS --fail-with-body -X DELETE "$API_BASE_URL/serverless/endpoints/$ENDPOINT_ID" \
    -H "$AUTH" >/dev/null
  echo "deleted $ENDPOINT_ID"
}
trap delete_endpoint EXIT