Skip to content
Hayonix

Quickstarts

Cache models on hosts

Pin a model from the model hub to a commit, mount it into serverless workers and pods from the host’s model cache, and load it with networking off, so a worker on a host that already holds the model starts without downloading it.

Most of a model worker's cold start is often the model download, not the container. Instead of baking weights into your image or downloading them in your handler, name the model in the workload: the host fetches it once into its model cache, checks every file's size and digest, and mounts it read-only into each container that asks for it. A later worker on a host that already holds the model starts without downloading anything.

You name a model through a model reference: a repository on the model hub, pinned to one commit. Serverless releases and pods list references in models.

Accept a model reference

Give the model as owner/name or owner/name@revision, where the revision is a branch, a tag or a commit (default main). The API checks the reference against the model hub before it answers (a few seconds): the repository exists, the revision resolves, and the files are listed with their sizes and digests. A branch is pinned to the commit it points at now, so a later push to the branch never changes what your workers run; accept a new reference to move to a newer commit. Nothing is downloaded yet.

include keeps only the files that match one of its patterns (* matches any characters, / included; ? one character), so a repository that holds several formats or quantisations is fetched only in the one you use.

REF=$(
  jq -n --arg model "$MODEL_REF" '{model: $model, include: ["*.json", "*.txt"]}' |
  curl -sS --fail-with-body "$API_BASE_URL/model-references" \
    -H "$AUTH" -H "Content-Type: application/json" -d @-
)
REF_ID=$(echo "$REF" | jq -r .id)
echo "$REF" | jq -r '"reference \(.id) \(.repository) at \(.commit[0:7])",
  "files: \(.file_count) bytes: \(.total_bytes) visibility: \(.visibility)"'

The reference records what it resolved to: commit (40 hex characters), files with each one's size and digest, file_count, total_bytes, and snapshot_path, the directory that holds the files inside a container.

Error codeMeaning
invalid_modelmodel is not owner/name or owner/name@revision
model_not_foundno such repository, or it is private and no token was given
revision_not_foundthe repository has no such branch, tag or commit
model_access_deniedthe repository is gated or private and the token cannot read it
no_matching_filesno file of the revision matches include
unsupported_file_namea file name cannot be cached (letters, digits, ., _ and - only); leave it out with include
model_too_largethe selected files exceed the size a reference may have; narrow it with include
unknown_secrettoken_secret_id is not a secret of this project
model_hub_unavailablethe model hub did not answer in time: retry

Private and gated models

For a private or gated repository, store an access token for the model hub as a project secret and pass its ID as token_secret_id. The reference is then visibility: private: hosts download it with that token and keep its files for this project only, so another project on the same host never mounts them, even for the same commit. Your containers never see the token. Public models are shared across projects on a host: the second project to mount one starts as fast as the first project's second worker.

Mount it in a serverless endpoint

List the reference in the release's models (up to 8, one revision per repository). Before a worker's container starts, its host makes each model present, downloading only what it does not hold yet, then mounts it read-only. Changing models makes a new release, rolled out like any other.

ENDPOINT_ID=$(
  echo "$REF" | jq --arg image "$IMAGE" --arg sku "$SKU" --arg region "$REGION" '{
    name: "model-reader", image: $image, skus: [$sku], region: $region, max_workers: 1,
    models: [{reference_id: .id}],
    env: [{name: "MODEL_REPOSITORY", value: .repository}, {name: "MODEL_COMMIT", value: .commit}]
  }' |
  curl -sS --fail-with-body "$API_BASE_URL/serverless/endpoints" \
    -H "$AUTH" -H "Content-Type: application/json" -d @- |
  jq -r .id
)
echo "endpoint $ENDPOINT_ID mounts $(echo "$REF" | jq -r .snapshot_path)"

This worker reads the repository and commit from environment variables set from the reference; a worker that always serves one model may hard-code them instead.

Load it with networking off

Inside the container each model appears at /model-cache/models--<owner>--<name>, laid out the way the model hub's client libraries keep their cache (the files of the commit under snapshots/<commit>/). HF_HUB_CACHE is set to /model-cache unless your environment sets HF_HUB_CACHE or HF_HOME itself, so the client libraries find the model without being told where it is.

Set HF_HUB_OFFLINE=1 and ask for the reference's commit with local_files_only=True: the call resolves to the mounted directory, and fails at once rather than downloading if the model is missing. Loading with from_pretrained works the same way with HF_HUB_OFFLINE=1 set.

handler.py
import json
import os

# The model is already on the host and mounted read-only: never reach for the network.
os.environ.setdefault("HF_HUB_OFFLINE", "1")

import serverless_worker as sw  # noqa: E402
from huggingface_hub import snapshot_download  # noqa: E402

# Set on the endpoint from the model reference (its repository and the commit it is pinned to).
REPOSITORY = os.environ["MODEL_REPOSITORY"]
COMMIT = os.environ["MODEL_COMMIT"]

# Once per worker, at start: HF_HUB_CACHE points at the mounted cache, so this only resolves a
# path. With local_files_only it fails loudly instead of downloading if the model is missing.
path = snapshot_download(REPOSITORY, revision=COMMIT, local_files_only=True)
with open(os.path.join(path, "config.json")) as f:
    config = json.load(f)


def handler(job: sw.Job):
    """Answers from the loaded model: here, what its configuration says."""
    return {
        "repository": REPOSITORY,
        "commit": COMMIT,
        "model_type": config.get("model_type"),
        "files": sorted(os.listdir(path)),
    }


if __name__ == "__main__":
    sw.serve(handler)

The image needs the model hub client, not the model:

Dockerfile
# The worker base image plus the model hub client. The model itself is not in the image: hosts
# cache it and mount it into the container before the worker starts.
ARG BASE=registry.example.com/team/serverless-worker:0.1.0
FROM ${BASE}

USER root
RUN pip install --no-cache-dir huggingface_hub==2.4.0
USER 10001

COPY handler.py /app/handler.py
CMD ["python", "/app/handler.py"]
Build and push
IMAGE=registry.example.com/team/model-reader:1.0.0
docker build -t "$IMAGE" .
docker push "$IMAGE"

The mount is read-only

Write anything derived from the model (compiled kernels, converted weights) to the container disk or a volume, not under /model-cache.

Run a job

The first job on a new endpoint waits for a worker to start, which now includes the host fetching the model unless it already holds it.

JOB_ID=$(
  curl -sS --fail-with-body "$API_BASE_URL/serverless/endpoints/$ENDPOINT_ID/jobs" \
    -H "$AUTH" -H "Content-Type: application/json" -d '{"input": {}}' |
  jq -r .id
)
# Long-poll until the job finishes (the first job waits for a worker to start).
for _ in $(seq 15); do
  JOB=$(curl -sS --fail-with-body "$API_BASE_URL/serverless/jobs/$JOB_ID?wait=60" -H "$AUTH")
  case "$(echo "$JOB" | jq -r .state)" in queued | running) ;; *) break ;; esac
done
echo "$JOB" | jq -r '"state: \(.state) model_type: \(.output.model_type)"'

See how a worker started

Each worker lists the steps of its start in start, in order: the image, each model, the container. The console's Workers tab shows the same steps, live while a worker starts. Loading inside the container ends when the worker is ready (ready_at).

curl -sS --fail-with-body "$API_BASE_URL/serverless/endpoints/$ENDPOINT_ID/workers" -H "$AUTH" |
jq -r '.data[].start[] |
  if .step == "model"
  then "model \(.repository) \(.state) downloaded bytes: \(.bytes_downloaded // 0)"
  else "\(.step) \(.state)" end'
StepStates
imagepulling, pulled, present (already on the host), failed
modelcached (found in the host's model cache: nothing downloaded), downloading (with bytes_done of bytes_total), downloaded (with bytes_downloaded and seconds), failed
containerstarted

bytes_downloaded counts what this fetch received from the network: 0 when everything was cached, less than bytes_total when part of the model was already there or an interrupted download was resumed.

Pods

Pods take models too, on create and on reconfigure, and see the same mount and HF_HUB_CACHE. Changing a pod's models recreates its container: only volumes survive.

Delete it

A reference can be deleted once nothing mounts it: no live workload, and no endpoint whose current release names it (409 with code model_reference_in_use until then). Hosts drop its files as their caches fill.

delete_all() {
  curl -sS --fail-with-body -X DELETE "$API_BASE_URL/serverless/endpoints/$ENDPOINT_ID" \
    -H "$AUTH" >/dev/null
  # A reference can be deleted once nothing mounts it (409 until the worker has stopped).
  for _ in $(seq 60); do
    CODE=$(curl -sS -o /dev/null -w '%{http_code}' -X DELETE \
      "$API_BASE_URL/model-references/$REF_ID" -H "$AUTH")
    [ "$CODE" = 200 ] && break
    [ "$CODE" = 409 ] || { echo "deleting $REF_ID: HTTP $CODE"; return 1; }
    sleep 2
  done
  echo "deleted $ENDPOINT_ID and $REF_ID"
}
trap delete_all EXIT

How the cache behaves

  • Each host's cache has a budget. When a new model needs the room, the models no running or paused workload mounts are evicted, least recently used first; a mounted file is never removed. If that frees too little, the fetch fails with model_cache_full and is retried later.
  • New work is placed, where the hardware allows, on a host that already holds its models.
  • Files are content-addressed and verified: a file that does not match its recorded size and digest is never mounted.
  • A host may be warmed ahead of a launch by the platform; that changes how fast a worker starts, never what it runs.

Next