Quickstarts
Cache models on hosts
Pin a model from the model hub to a commit, mount it into serverless workers and pods from the host’s model cache, and load it with networking off, so a worker on a host that already holds the model starts without downloading it.
Most of a model worker's cold start is often the model download, not the container. Instead of baking weights into your image or downloading them in your handler, name the model in the workload: the host fetches it once into its model cache, checks every file's size and digest, and mounts it read-only into each container that asks for it. A later worker on a host that already holds the model starts without downloading anything.
You name a model through a model reference: a repository on the model hub, pinned to one commit. Serverless releases and pods list references in models.
Accept a model reference
Give the model as owner/name or owner/name@revision, where the revision is a branch, a tag or a commit (default main). The API checks the reference against the model hub before it answers (a few seconds): the repository exists, the revision resolves, and the files are listed with their sizes and digests. A branch is pinned to the commit it points at now, so a later push to the branch never changes what your workers run; accept a new reference to move to a newer commit. Nothing is downloaded yet.
include keeps only the files that match one of its patterns (* matches any characters, / included; ? one character), so a repository that holds several formats or quantisations is fetched only in the one you use.
REF=$(
jq -n --arg model "$MODEL_REF" '{model: $model, include: ["*.json", "*.txt"]}' |
curl -sS --fail-with-body "$API_BASE_URL/model-references" \
-H "$AUTH" -H "Content-Type: application/json" -d @-
)
REF_ID=$(echo "$REF" | jq -r .id)
echo "$REF" | jq -r '"reference \(.id) \(.repository) at \(.commit[0:7])",
"files: \(.file_count) bytes: \(.total_bytes) visibility: \(.visibility)"'The reference records what it resolved to: commit (40 hex characters), files with each one's size and digest, file_count, total_bytes, and snapshot_path, the directory that holds the files inside a container.
| Error code | Meaning |
|---|---|
invalid_model | model is not owner/name or owner/name@revision |
model_not_found | no such repository, or it is private and no token was given |
revision_not_found | the repository has no such branch, tag or commit |
model_access_denied | the repository is gated or private and the token cannot read it |
no_matching_files | no file of the revision matches include |
unsupported_file_name | a file name cannot be cached (letters, digits, ., _ and - only); leave it out with include |
model_too_large | the selected files exceed the size a reference may have; narrow it with include |
unknown_secret | token_secret_id is not a secret of this project |
model_hub_unavailable | the model hub did not answer in time: retry |
Private and gated models
For a private or gated repository, store an access token for the model hub as a project secret and pass its ID as token_secret_id. The reference is then visibility: private: hosts download it with that token and keep its files for this project only, so another project on the same host never mounts them, even for the same commit. Your containers never see the token. Public models are shared across projects on a host: the second project to mount one starts as fast as the first project's second worker.
Mount it in a serverless endpoint
List the reference in the release's models (up to 8, one revision per repository). Before a worker's container starts, its host makes each model present, downloading only what it does not hold yet, then mounts it read-only. Changing models makes a new release, rolled out like any other.
ENDPOINT_ID=$(
echo "$REF" | jq --arg image "$IMAGE" --arg sku "$SKU" --arg region "$REGION" '{
name: "model-reader", image: $image, skus: [$sku], region: $region, max_workers: 1,
models: [{reference_id: .id}],
env: [{name: "MODEL_REPOSITORY", value: .repository}, {name: "MODEL_COMMIT", value: .commit}]
}' |
curl -sS --fail-with-body "$API_BASE_URL/serverless/endpoints" \
-H "$AUTH" -H "Content-Type: application/json" -d @- |
jq -r .id
)
echo "endpoint $ENDPOINT_ID mounts $(echo "$REF" | jq -r .snapshot_path)"This worker reads the repository and commit from environment variables set from the reference; a worker that always serves one model may hard-code them instead.
Load it with networking off
Inside the container each model appears at /model-cache/models--<owner>--<name>, laid out the way the model hub's client libraries keep their cache (the files of the commit under snapshots/<commit>/). HF_HUB_CACHE is set to /model-cache unless your environment sets HF_HUB_CACHE or HF_HOME itself, so the client libraries find the model without being told where it is.
Set HF_HUB_OFFLINE=1 and ask for the reference's commit with local_files_only=True: the call resolves to the mounted directory, and fails at once rather than downloading if the model is missing. Loading with from_pretrained works the same way with HF_HUB_OFFLINE=1 set.
import json
import os
# The model is already on the host and mounted read-only: never reach for the network.
os.environ.setdefault("HF_HUB_OFFLINE", "1")
import serverless_worker as sw # noqa: E402
from huggingface_hub import snapshot_download # noqa: E402
# Set on the endpoint from the model reference (its repository and the commit it is pinned to).
REPOSITORY = os.environ["MODEL_REPOSITORY"]
COMMIT = os.environ["MODEL_COMMIT"]
# Once per worker, at start: HF_HUB_CACHE points at the mounted cache, so this only resolves a
# path. With local_files_only it fails loudly instead of downloading if the model is missing.
path = snapshot_download(REPOSITORY, revision=COMMIT, local_files_only=True)
with open(os.path.join(path, "config.json")) as f:
config = json.load(f)
def handler(job: sw.Job):
"""Answers from the loaded model: here, what its configuration says."""
return {
"repository": REPOSITORY,
"commit": COMMIT,
"model_type": config.get("model_type"),
"files": sorted(os.listdir(path)),
}
if __name__ == "__main__":
sw.serve(handler)The image needs the model hub client, not the model:
# The worker base image plus the model hub client. The model itself is not in the image: hosts
# cache it and mount it into the container before the worker starts.
ARG BASE=registry.example.com/team/serverless-worker:0.1.0
FROM ${BASE}
USER root
RUN pip install --no-cache-dir huggingface_hub==2.4.0
USER 10001
COPY handler.py /app/handler.py
CMD ["python", "/app/handler.py"]IMAGE=registry.example.com/team/model-reader:1.0.0
docker build -t "$IMAGE" .
docker push "$IMAGE"The mount is read-only
Write anything derived from the model (compiled kernels, converted weights) to the container disk or a volume, not under /model-cache.
Run a job
The first job on a new endpoint waits for a worker to start, which now includes the host fetching the model unless it already holds it.
JOB_ID=$(
curl -sS --fail-with-body "$API_BASE_URL/serverless/endpoints/$ENDPOINT_ID/jobs" \
-H "$AUTH" -H "Content-Type: application/json" -d '{"input": {}}' |
jq -r .id
)
# Long-poll until the job finishes (the first job waits for a worker to start).
for _ in $(seq 15); do
JOB=$(curl -sS --fail-with-body "$API_BASE_URL/serverless/jobs/$JOB_ID?wait=60" -H "$AUTH")
case "$(echo "$JOB" | jq -r .state)" in queued | running) ;; *) break ;; esac
done
echo "$JOB" | jq -r '"state: \(.state) model_type: \(.output.model_type)"'See how a worker started
Each worker lists the steps of its start in start, in order: the image, each model, the container. The console's Workers tab shows the same steps, live while a worker starts. Loading inside the container ends when the worker is ready (ready_at).
curl -sS --fail-with-body "$API_BASE_URL/serverless/endpoints/$ENDPOINT_ID/workers" -H "$AUTH" |
jq -r '.data[].start[] |
if .step == "model"
then "model \(.repository) \(.state) downloaded bytes: \(.bytes_downloaded // 0)"
else "\(.step) \(.state)" end'| Step | States |
|---|---|
image | pulling, pulled, present (already on the host), failed |
model | cached (found in the host's model cache: nothing downloaded), downloading (with bytes_done of bytes_total), downloaded (with bytes_downloaded and seconds), failed |
container | started |
bytes_downloaded counts what this fetch received from the network: 0 when everything was cached, less than bytes_total when part of the model was already there or an interrupted download was resumed.
Pods
Pods take models too, on create and on reconfigure, and see the same mount and HF_HUB_CACHE. Changing a pod's models recreates its container: only volumes survive.
Delete it
A reference can be deleted once nothing mounts it: no live workload, and no endpoint whose current release names it (409 with code model_reference_in_use until then). Hosts drop its files as their caches fill.
delete_all() {
curl -sS --fail-with-body -X DELETE "$API_BASE_URL/serverless/endpoints/$ENDPOINT_ID" \
-H "$AUTH" >/dev/null
# A reference can be deleted once nothing mounts it (409 until the worker has stopped).
for _ in $(seq 60); do
CODE=$(curl -sS -o /dev/null -w '%{http_code}' -X DELETE \
"$API_BASE_URL/model-references/$REF_ID" -H "$AUTH")
[ "$CODE" = 200 ] && break
[ "$CODE" = 409 ] || { echo "deleting $REF_ID: HTTP $CODE"; return 1; }
sleep 2
done
echo "deleted $ENDPOINT_ID and $REF_ID"
}
trap delete_all EXITHow the cache behaves
- Each host's cache has a budget. When a new model needs the room, the models no running or paused workload mounts are evicted, least recently used first; a mounted file is never removed. If that frees too little, the fetch fails with
model_cache_fulland is retried later. - New work is placed, where the hardware allows, on a host that already holds its models.
- Files are content-addressed and verified: a file that does not match its recorded size and digest is never mounted.
- A host may be warmed ahead of a launch by the platform; that changes how fast a worker starts, never what it runs.
Next
- Your first serverless endpoint: the worker runtime and the job lifecycle.
- Serve HTTP from a serverless endpoint: the same
modelswork for HTTP endpoints, and fast resume keeps a loaded model warm.