fleet

Add a machine from how the user connects to it: an ssh command such as `ssh -p 40001 root@1.2.3.4`, or a name from discover_machines. Ask for it if the user did not give one. If the machine refuses, the result has `why` and `fix` -- the one thing the user does to let fleet in. Never ask for a password.

Machines the user can already reach that are not added yet: the Host entries of their ~/.ssh/config, and the machines on their tailnet. Offer the ones that look relevant; add one with add_machine(ssh_command=its name) once the user agrees.

Why a machine will not let fleet in, and the exact fix for it: the key line to run there, the provider account to add the key to, or the Tailscale SSH rule. Changes nothing; tell the user what it says -- access is theirs to grant.

Label a machine, or change what it costs per hour (for the $/HR column and burn rate), how it is reached, or which disks to watch for free space: disk_paths means exactly these paths from now on -- for a container or rental whose `/` is not where the space is -- and autodetect_disks goes back. rename gives it a new name, alias a short one (empty clears it). Tags, cost and note are written to the machine's own machine.yaml, so everyone who looks at it reads them; local=True keeps tags and cost in this machine's list instead (for a machine you cannot or should not write to). Measured facts such as cuda or vram-24g come from probes.

Every job on every GPU across the fleet -- who runs it, how much VRAM, for how long, the command -- and which GPUs are idle. A GPU with jobs is someone's: ask before using it, and so is one in `busy_unseen`, in use by processes this login cannot see.

Every machine, with what is free right now. Filter with `tag` to pick by capability -- repeated tags must all match. Facts include gpu, cuda, rocm, metal, multi-gpu, vram-NNg, linux/macos/windows, x86_64/arm64, cores-NN, ram-NNg, storage-NNt, public-ip/mesh/lan, rental/shared/appliance, plus any tag a human set.

Every model being served across the fleet: machine, server, url, and for a server that only listens on its own machine a `tunnel` command to reach it (`reach: down`: declared but not running); plus the folders each machine keeps models in, and `not_read`: machines that did not answer, so their servers are not known. Check this before downloading a model: one already served or on disk need not be fetched. `models` names a server's first 20; `models_total` says how many it serves, and the rest are at its own /v1/models.

Run one command on a machine and return its output and exit code. This is how work actually gets started -- fleet resolves the address and connects as the user does, so never build an ssh command by hand. Returns within 120 seconds: start long jobs detached (nohup ... > log 2>&1 &).

Declare a model server the probe would not find by itself (an unusual port, a program it does not recognise), in the machine's own machine.yaml: port, and a base path if it is not /v1. port=None removes it (a name not declared comes back in `not_declared`). Never pass an API key; a server that wants one is shown as wanting one.

Record where a machine keeps things, in its own machine.yaml: kind is models, datasets, media, scratch, envs or one of your own; paths replace that kind's folders, and an empty list clears it. Ask the user where before recording a folder they did not name.

Full detail for one machine: CPU, RAM, every GPU and its jobs, mounts, listening services, the models its servers serve, its software (cuda, python, and each Python environment's torch/vllm/mlx/... versions), the folders it keeps models, datasets and media in, tags and facts. No name means the machine fleet is running on.