Skip to content
Hayonix

Quickstarts

Run a multi-node job on a cluster

Create identical members on one private network across hosts, all at once or not at all, reach the head over SSH, start processes on every member with the launcher, run torchrun or mpirun across them, and share files on a shared volume.

A cluster is a fixed number of identical members — same image, same hardware — placed on one private network that spans hosts, for one distributed job: multi-node training, an MPI program, a parameter sweep. Member 0 is the head. Every member gets a fixed private address and a hostname, member-<rank> (the head is also head), that resolves on every member, and the environment variables that launchers such as torchrun and mpirun need. An optional shared volume is mounted read-write at the same path on every member.

Before you start

Environment variables
VariableValue
API_KEYsk_… (your project API key)
API_BASE_URLhttps://api.hayonix.ai/v1
INFERENCE_BASE_URLhttps://api.hayonix.ai/v1

The samples read IMAGE, SKU and REGION. A cluster has 2 to 16 members. The organization's clusters, cluster_members and gpus quotas apply (see Quotas and limits). An API key needs the cluster:* scopes.

What it costs

GET /v1/clusters/estimate prices a shape at today's prices. Each member is billed per second from the start of its run, like a pod, at its hardware's price, until its container stops; a member waiting to start is not billed. The shared volume is billed per GB-second for as long as the cluster exists, until it is deleted. See How compute is billed.

curl -sS --fail-with-body -G "$API_BASE_URL/clusters/estimate" -H "$AUTH" \
  --data-urlencode "sku=$SKU" -d member_count=2 -d shared_volume_gb=10 |
  jq -r '"total per hour (nano-USD): \(.total_per_hour)"'

Create a cluster

Every member is placed on a host before the request returns, or nothing is: if any member does not fit — not enough free hosts, CPUs, memory or GPUs in the region — the request fails with 409 and code insufficient_capacity, and nothing is created or billed. Clusters never preempt other work. Members are spread over hosts first, and may share a host when the region has room for several on one.

# Every member runs the same command; this one prints where it is and stays up.
COMMAND='echo "rank=$CLUSTER_RUNTIME_RANK of $CLUSTER_RUNTIME_SIZE host=$CLUSTER_RUNTIME_HOSTNAME head=$CLUSTER_RUNTIME_HEAD_ADDRESS"; exec sleep infinity'
if ! CLUSTER=$(
  jq -n --arg image "$IMAGE" --arg sku "$SKU" --arg region "$REGION" --arg cmd "$COMMAND" '{
    name: "guide",
    member_count: 2,
    hardware: { sku: $sku, gpu_count: 0 },
    region: $region,
    image: $image,
    command: ["sh", "-c", $cmd],
    shared_volume: { size_gb: 10, mount_path: "/shared" }
  }' |
    curl -sS --fail-with-body "$API_BASE_URL/clusters" \
      -H "$AUTH" -H "Content-Type: application/json" -d @-
); then
  # All or nothing: a 409 insufficient_capacity means nothing was created or billed.
  echo "$CLUSTER" | jq -r '"not created: \(.error.code): \(.error.message)"'
  exit 1
fi
CLUSTER_ID=$(echo "$CLUSTER" | jq -r .id)
echo "created $CLUSTER_ID $(echo "$CLUSTER" | jq -r .state)"

command replaces the image's entrypoint on every member; an image whose command exits makes its member failed, so for interactive work keep the members up (sleep infinity, a notebook, an SSH-ready image) and start your job from the head.

A cluster's configuration never changes after creation: only its name can (PATCH /v1/clusters/{cluster_id}).

Wait until every member runs

The cluster is provisioning while members pull their image and start, and running once every member's container runs. GET /v1/clusters/{cluster_id} lists every member with its rank, hostname, private address, state and host (host_id is opaque; members with the same value share a host), and the ready-to-run ssh command for each.

# provisioning → running once every member runs; degraded means one failed (nothing restarts it).
while STATE=$(curl -sS --fail-with-body "$API_BASE_URL/clusters/$CLUSTER_ID" -H "$AUTH" | jq -r .state)
  [ "$STATE" = provisioning ]; do sleep 3; done
[ "$STATE" = running ] || { echo "cluster is $STATE"; exit 1; }
curl -sS --fail-with-body "$API_BASE_URL/clusters/$CLUSTER_ID" -H "$AUTH" |
  jq -r '.members[] | "member \(.rank): \(.hostname) \(.address) \(.state)",
    (select(.rank == 0) | "ssh to the head: \(.ssh.command)")'

Connect to the head

Members are reached through the platform's SSH gateway with one of the project's SSH keys (hayonix ssh-keys add ~/.ssh/id_ed25519.pub, or the console). Run the head's ssh.command; hayonix clusters ssh <cluster> prints it (--rank N for another member, --known-hosts for lines that pin the gateway's host keys). Nothing in the image needs to run an SSH server.

The rank and host environment

Every member has these variables, set by the platform (customers cannot set names starting with CLUSTER_RUNTIME_):

VariableValue
CLUSTER_RUNTIME_CLUSTER_IDThe cluster's ID
CLUSTER_RUNTIME_RANKThis member's rank, 0 on the head
CLUSTER_RUNTIME_SIZENumber of members
CLUSTER_RUNTIME_HOSTNAME, CLUSTER_RUNTIME_ADDRESSThis member's hostname (member-<rank>) and private address
CLUSTER_RUNTIME_HEAD_HOSTNAME, CLUSTER_RUNTIME_HEAD_ADDRESSThe head's
CLUSTER_RUNTIME_HOSTNAMESEvery hostname in rank order, separated by spaces
CLUSTER_RUNTIME_HOSTSmember-0=<address>,member-1=<address>,… in rank order
CLUSTER_RUNTIME_GPUS_PER_MEMBERGPUs on each member (0 on CPU hardware)
CLUSTER_RUNTIME_LAUNCHERThe launcher (below)
CLUSTER_RUNTIME_SHARED_PATHWhere the shared volume is mounted, when there is one
On a member
# (CLUSTER_RUNTIME_SSH_KEY is the key members use to reach each other: keep it out of logs.)
env | grep '^CLUSTER_RUNTIME_' | grep -v '^CLUSTER_RUNTIME_SSH_KEY=' | sort
# CLUSTER_RUNTIME_ADDRESS=10.224.5.2
# CLUSTER_RUNTIME_CLUSTER_ID=clu_01jc…
# CLUSTER_RUNTIME_GPUS_PER_MEMBER=0
# CLUSTER_RUNTIME_HEAD_ADDRESS=10.224.5.2
# CLUSTER_RUNTIME_HEAD_HOSTNAME=member-0
# CLUSTER_RUNTIME_HOSTNAME=member-0
# CLUSTER_RUNTIME_HOSTNAMES=member-0 member-1
# CLUSTER_RUNTIME_HOSTS=member-0=10.224.5.2,member-1=10.224.5.3
# CLUSTER_RUNTIME_LAUNCHER=/.nodeagent/member-ssh
# CLUSTER_RUNTIME_RANK=0
# CLUSTER_RUNTIME_SHARED_PATH=/shared
# CLUSTER_RUNTIME_SIZE=2

Addresses and hostnames are fixed for the cluster's life. On the private network a member reaches the cluster's own members and nothing else: not other clusters' members, not other projects' pods, not the hosts.

Run a command on every member

$CLUSTER_RUNTIME_LAUNCHER starts a process on another member. It takes ssh's arguments — a host, then the command — and returns the command's output and exit code as ssh would, so it works wherever a tool expects an ssh-compatible launcher. Nothing has to be installed, no keys exchanged, no SSH server configured.

On the head
# The launcher takes ssh's arguments: a host, then the command. Its exit code and stderr come back
# as ssh would return them. Nothing to install or configure in the image.
$CLUSTER_RUNTIME_LAUNCHER member-1 hostname
for h in $CLUSTER_RUNTIME_HOSTNAMES; do
  $CLUSTER_RUNTIME_LAUNCHER "$h" "hostname; nproc"
done

Multi-node PyTorch with torchrun

Start one torchrun per member: --nnodes=$CLUSTER_RUNTIME_SIZE, --node_rank the member's rank, and the rendezvous on the head with --master_addr=$CLUSTER_RUNTIME_HEAD_ADDRESS. Use the gloo backend on CPU members and nccl on GPU members. Create the cluster with GLOO_SOCKET_IFNAME=eth0 (and NCCL_SOCKET_IFNAME=eth0 on GPUs) in its env, so the backends use the private network's interface. The image must have PyTorch.

The training script, on the shared volume
# On the shared volume, so every member runs the same file.
cat > "$CLUSTER_RUNTIME_SHARED_PATH/ddp.py" <<'EOF'
import os
import socket

import torch
import torch.distributed as dist

gpu = torch.cuda.is_available()
if gpu:
    torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))
dist.init_process_group("nccl" if gpu else "gloo")
rank, world = dist.get_rank(), dist.get_world_size()
t = torch.tensor([float(rank + 1)], device="cuda" if gpu else "cpu")
dist.all_reduce(t)
print(f"rank={rank} world={world} host={socket.gethostname()} sum={t.item():.0f}", flush=True)
dist.destroy_process_group()
EOF
On the head
# One torchrun per member: --node_rank is the member's rank, the rendezvous is on the head.
NPROC=$((CLUSTER_RUNTIME_GPUS_PER_MEMBER > 0 ? CLUSTER_RUNTIME_GPUS_PER_MEMBER : 1))
run_member() { # run_member RANK: the torchrun command line for that member
  echo "cd $CLUSTER_RUNTIME_SHARED_PATH && torchrun --nnodes=$CLUSTER_RUNTIME_SIZE" \
    "--node_rank=$1 --nproc_per_node=$NPROC" \
    "--master_addr=$CLUSTER_RUNTIME_HEAD_ADDRESS --master_port=29500 ddp.py"
}
# Every other member, in the background, through the launcher (output kept on the shared volume)…
for R in $(seq 1 $((CLUSTER_RUNTIME_SIZE - 1))); do
  $CLUSTER_RUNTIME_LAUNCHER "member-$R" "$(run_member "$R")" \
    > "$CLUSTER_RUNTIME_SHARED_PATH/rank-$R.log" 2>&1 &
done
# …and the head itself, in the foreground.
sh -c "$(run_member "$CLUSTER_RUNTIME_RANK")"
wait
cat "$CLUSTER_RUNTIME_SHARED_PATH"/rank-*.log
# rank=0 world=2 host=member-0 sum=3
# rank=1 world=2 host=member-1 sum=3

MPI with mpirun

mpirun starts its daemons on other hosts over ssh; point it at the launcher instead with --mca plm_rsh_agent $CLUSTER_RUNTIME_LAUNCHER, and give it a hostfile built from $CLUSTER_RUNTIME_HOSTNAMES. Every member must have the same MPI build, which it does when the image has one.

On the head
# A hostfile from the member list: one line per member, one slot per GPU (or one on CPU).
SLOTS=$((CLUSTER_RUNTIME_GPUS_PER_MEMBER > 0 ? CLUSTER_RUNTIME_GPUS_PER_MEMBER : 1))
for h in $CLUSTER_RUNTIME_HOSTNAMES; do echo "$h slots=$SLOTS"; done > hostfile

# mpirun starts its daemons on the other members through the launcher instead of ssh.
mpirun --mca plm_rsh_agent "$CLUSTER_RUNTIME_LAUNCHER" --hostfile hostfile \
  -np $((CLUSTER_RUNTIME_SIZE * SLOTS)) hostname

The shared volume

With shared_volume, one volume is mounted read-write at mount_path (default /shared) on every member, with the same files everywhere: a place for code, datasets, checkpoints and logs that every rank reads.

On the head
# The same files on every member: write on one, read on another.
echo "written on $CLUSTER_RUNTIME_HOSTNAME" > "$CLUSTER_RUNTIME_SHARED_PATH/hello.txt"
$CLUSTER_RUNTIME_LAUNCHER member-1 cat "$CLUSTER_RUNTIME_SHARED_PATH/hello.txt"

Degraded clusters

A member whose container exits or cannot start becomes failed; one whose host stops answering becomes host_lost. Either makes the cluster degraded (state_reason member_failed or host_lost). Nothing is restarted or replaced: the other members keep running — and billing — until you delete the cluster, because a distributed job has usually lost its group by then anyway. A member on a lost host stops being billed at the last second its host proved it was running. The timeline says which member and why.

curl -sS --fail-with-body "$API_BASE_URL/clusters/$CLUSTER_ID/events" -H "$AUTH" |
  jq -r '.data[] | "event: \(.type) \(.message)"'

A cluster can also be stopped by the platform (the organization ran out of credit, or a host was drained for maintenance): every member stops and only deletion remains.

A member's output

GET /v1/clusters/{cluster_id}/members/{rank}/logs streams a member's container output as server-sent events (output, notice when there is none to give, then end); follow=true keeps it open. Output comes from the member's host and is gone once its container is removed.

# A member's container output as server-sent events, from its host (?follow=true keeps it open).
LINE=""
for _ in 1 2 3 4 5 6 7 8 9 10; do
  LINE=$(curl -sS --fail-with-body "$API_BASE_URL/clusters/$CLUSTER_ID/members/1/logs?tail=20" \
    -H "$AUTH" | sed -n 's/^data: //p' | jq -r 'select(.text) | .text' | sed -n 1p)
  [ -n "$LINE" ] && break
  sleep 2
done
echo "member 1 says: $LINE"

Delete it

Deleting is irreversible: every member is stopped and removed, billing ends with each member's container, the shared volume's data is destroyed and the network is freed.

# Irreversible: every member, the network and the shared volume's data are destroyed.
delete_cluster() {
  curl -sS --fail-with-body -X DELETE "$API_BASE_URL/clusters/$CLUSTER_ID" -H "$AUTH" >/dev/null
  echo "deleted $CLUSTER_ID"
}
trap delete_cluster EXIT

Known limits

  • No resize, no reconfigure. Members, hardware, image and network are fixed when the cluster is created; create a new cluster for a new shape.
  • No stop and start. A cluster runs until it is deleted (or stopped by the platform, after which only deletion remains).
  • Members are never moved or restarted, on failure, on host loss or on host maintenance.
  • GPU members and high-speed fabrics are unverified. Clusters have run on CPU members only (torchrun with gloo); GPU members and NCCL across hosts have not been exercised, nor has mpirun. Traffic between hosts goes over the encrypted network, not InfiniBand or RoCE.