GPU scale-swap runbook¶

You do not deploy a GPU app. You swap which one holds the card, by flipping committed replica counts in one commit.

In plain words: each graphics card runs one app at a time. To give a card to a different app, scale the current owner to 0 and the new owner to 1 in Git.

New to the stack? Read Local AI starter kit first.

The rule¶

GPU workloads are mutually-exclusive whole-card: time-slicing off, every pod requests whole nvidia.com/gpu cards, every Deployment uses strategy: Recreate. Never two pods on a card.

Two things make this safe:

The scheduler enforces it. A scaled-up pod sits Pending until the outgoing pod releases its card. Insufficient nvidia.com/gpu during a swap is normal.
ArgoCD selfHeal reverts kubectl scale. The committed value is the only real switch.

Card truth table¶

App	Cards	`replicas` in git	File
vLLM (Qwen3.8-27B FP8)	2	`1`	`my-apps/ai/vllm/deployment.yaml`
NInfer-3090 (archived, not deployed)	1	n/a	`my-apps/ai/_archive/ninfer/deployment.yaml`
ComfyUI	1	`0`	`my-apps/ai/comfyui/deployment.yaml`
SwarmUI	1	`0`	`my-apps/ai/swarmui/deployment.yaml`
llmfit (archived, not deployed)	1	n/a	`my-apps/ai/_archive/llmfit/`

vLLM owns both cards, so there is no spare GPU while it runs. Sum replicas × cards across active workloads; the total must not exceed two. Replica overrides in each kustomization.yaml are authoritative.

The procedure¶

About 10 minutes plus reconciliation.

flowchart LR
    E[Edit both replica counts<br/>in one commit] --> P[PR and merge] --> A[ArgoCD syncs]
    A --> O[Old pod stops,<br/>card freed] --> N[New pod: Pending, then Running]

Pick the target state from the truth table.
Edit outgoing and incoming replica counts in one commit.
Open a PR. Merge is the operator's call.
Wait for the outgoing pod to release the card. Do not "fix" a Pending incoming pod.
Verify:

kubectl -n vllm get pods
kubectl -n comfyui get pods
kubectl -n swarmui get pods
kubectl -n gpu-operator exec ds/nvidia-powerlimit -- nvidia-smi
curl -fsS https://vllm.vanillax.me/v1/models

Expect the incoming pod Running, two cards at 220 W, and the endpoint serving.

Both LAN hostnames route to vLLM. A swap changes replica ownership and service/route wiring together.

What breaks while vLLM is at 0¶

Open WebUI, Perplexica, SurfSense, LiteLLM, Presenton, Hermes, Home Assistant and any Pi session on the cluster endpoint lose their backend.

ComfyUI's vision-to-image helper needs the chat backend, so it cannot run alongside vLLM. Un-archived GPU jobs such as llmfit need the two-card server parked first.

Don'ts¶

Don't kubectl scale — selfHeal reverts it.
Don't set NVIDIA_VISIBLE_DEVICES or CUDA_VISIBLE_DEVICES in workload pods. The nvidia-powerlimit DaemonSet is the intentional exception.
Don't switch a GPU Deployment to RollingUpdate. Recreate releases the whole card cleanly and avoids RWO Multi-Attach.
Don't raise the 220 W cap to chase throughput. POWER_LIMIT_WATTS lives in infrastructure/gpu/nvidia-gpu-operator/powerlimit-daemonset.yaml; changing it is an electrical decision.