VPA policy ownership and scheduling

Purpose: explain how this cluster sizes pod requests with Vertical Pod Autoscaler (VPA), where each policy belongs, and how request resizing interacts with horizontal scaling and topology.

Status: current architecture and operator runbook.

Scope: VPA request recommendations/actuation, topology spread, and disruption budgets. A descheduler is deliberately not installed: eviction-based rebalancing needs a separate policy review because many workloads mount ReadWriteOnce volumes.

Mental model

VPA is three separate runtime actions:

1. The recommender learns per-container CPU and memory usage.
2. The updater applies InPlaceOrRecreate recommendations to existing pods.
3. The admission controller applies recommendations when new pods are created.

Stopping only the updater is therefore not a cluster-wide off switch. New pods can still be changed by admission. To stop all actuation while preserving learning, disable both updater.enabled and admissionController.enabled in infrastructure/controllers/vertical-pod-autoscaler/values.yaml; leave the recommender enabled.

VPA changes requests, not placement. The scheduler evaluates topology spread only when a pod is scheduled, and an in-place resize intentionally does not move the pod. A balanced workload can therefore become imbalanced as requests grow.

Repository layout

Owner	Policy location	Why
User application	my-apps/<category>/<app>/vpa.yaml	App deletion prunes its policy; target and sizing review happen together
Monitoring application	monitoring/<app>/vpa.yaml	Same ownership rule, including Helm-rendered and operator CR targets
Bootstrap/system workload	infrastructure/controllers/vpa-system-policies/	Explicit exception for workloads whose owner reconciles at the VPA CRD wave or otherwise cannot safely own the policy after CRD bootstrap
VPA controller	infrastructure/controllers/vertical-pod-autoscaler/	Recommender, updater, admission controller, and CRDs at wave 4
VPA monitoring	infrastructure/controllers/vertical-pod-autoscaler-observability/	Optional PodMonitor and alerts at wave 6, after Prometheus CRDs

The policy object identity is its namespace and name. Keep both unchanged when moving a policy between Argo applications. The recommender's checkpoints and in-memory aggregate histories can then reattach after the ownership handoff. During the one-PR migration, the historical central Application reconciles at wave 4 and prunes moved policies before monitoring owners at wave 5 and user application owners at wave 6 adopt them. PostHog is a temporary exception: because its migration hook currently blocks sync, its policies stay central until that Application is sync-healthy. Two rendered policies targeting the same workload are never permitted.

The system-policy entrypoint intentionally retains the historical Argo Application name vpa-recommendations. Its file and source path describe the new responsibility; preserving the Application identity prevents a cascading delete of every existing VPA. It carries SkipDryRunOnMissingResource=true and retry settings because it shares wave 4 with the VPA controller on a cold bootstrap. Infrastructure applications at that same wave remain central unless they gain an explicitly ordered post-CRD owner.

Policy contract

Active policies use:

updatePolicy:
  updateMode: InPlaceOrRecreate
  minReplicas: 1
resourcePolicy:
  containerPolicies:
    - containerName: app
      controlledResources: ["cpu", "memory"]
      controlledValues: RequestsOnly
      maxAllowed:
        cpu: "2"
        memory: 4Gi

The rules behind that shape are:

• RequestsOnly leaves Git-declared limits as the hard runtime boundary.
• minReplicas: 1 is required because this cluster is mostly single replica.
• maxAllowed is per container, not per pod. A wildcard 2 GiB ceiling on a three-container pod permits up to 6 GiB. Multi-container workloads therefore name their workload containers and use mode: Off for fixed-size sidecars.
• A CPU-utilization HPA divides usage by the CPU request. A VPA must not control CPU on that same target because it moves the HPA denominator. The supported pairing is CPU HPA plus memory-only VPA.
• Off is reserved for learn-only policies such as a scale-swap workload.

The rendered-manifest check in scripts/validate-vpa-policies.py rejects missing targets, duplicate targets, unsupported modes, unsafe CPU HPA overlap, and active policies without the safety fields. It warns about uncovered workloads and wildcard ceilings. Reviewed exclusions live in scripts/vpa-exemptions.yaml; prose-only exclusions are not policy.

Topology and disruption

The active Omni template uses physical-host zone labels:

Physical host	Talos nodes	Zone
HP SFF	Control plane and worker	hp-sff
HP Elite	Worker	hp-elite
Dell	Temporary worker	dell
Threadripper	GPU worker	house
Shed HP	Worker behind the media bridge	shed

Cloudflared uses soft ScheduleAnyway spread and preferred anti-affinity over topology.kubernetes.io/zone, so two VMs on the SFF count as one physical host. Its preferred node affinity favors the general worker pool. These are preferences: other eligible workers remain usable when the HPs lack room, and several tunnel pods can share a surviving host. Verify actual placement; this is not a guarantee that every replica occupies a different chassis.

What each worker is for

The Omni template declares a separate node.vanillax.dev/pool label:

Worker	Pool	Intended job
HP SFF and HP Elite	general	Everyday services and selected application state
Threadripper	gpu	GPU workloads and selected heavy jobs
Shed HP Micro	edge	Attached devices and permitted edge jobs
Dell	disposable	Temporary, restartable work after required state is relocated

These labels describe intent; existing apps and Longhorn replicas are not moved by a label alone. In particular, the Dell still holds real data. Cloudflared is the first consumer and uses a soft preference. No new taints, hard app constraints, storage tags or control-plane scheduling changes accompany these labels.

Check live labels after syncing the Omni template; merging Git does not itself change allocated machines. Finish the active Talos upgrade before applying another template change. Confirm the Omni sync preview contains the expected label changes without machine replacement, then check:

kubectl get nodes -L node.vanillax.dev/pool,topology.kubernetes.io/zone
kubectl -n cloudflared get pods -o wide

Expected labels match the table, and all three tunnel replicas become Ready after the ordinary Deployment rollout. If the labels are not applied yet, preferred affinity still allows the tunnel pods to schedule. Rolling back the Deployment preference is independent of reverting the Omni labels. A label change is not a request to drain workers or re-provision disks.

Replicated critical workloads should pair spread with a PodDisruptionBudget using maxUnavailable: 1 and unhealthyPodEvictionPolicy: AlwaysAllow. Do not add minAvailable: 1 to a single-replica controller: it blocks voluntary Talos drains. The VPA controller currently runs one replica per component, so its PDBs intentionally remain off.

Topology spread is not a rebalance loop. If request growth repeatedly fills one node, first correct ceilings and placement constraints. Evaluate a descheduler only after classifying RWO workloads and defining which pods may be evicted.

The hardware and placement review proposes workload pools alongside these existing physical-host zones. Pool labels, new taints and placement migrations in that review are not deployed configuration.

Verification

Run from the repository root:

kustomize build infrastructure/controllers/vertical-pod-autoscaler --enable-helm
kustomize build infrastructure/controllers/vpa-system-policies
kustomize build infrastructure/controllers/vertical-pod-autoscaler-observability
python3 scripts/validate-vpa-policies.py /tmp/all-manifests.yaml

The last command expects the aggregate render produced by Cluster CI. Success reports one unique target per VPA and zero errors. Coverage warnings require either a new co-located policy or a reasoned machine-readable exemption.

After Argo syncs, verify:

kubectl get vpa -A
kubectl get vpa -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,MODE:.spec.updatePolicy.updateMode'
kubectl get nodes -L topology.kubernetes.io/zone
kubectl get pods -n cloudflared -o wide
kubectl get pdb -n cloudflared

Expected results are active policies in InPlaceOrRecreate, the learn-only policy in Off, zone values matching the table after the Omni template is synced, cloudflared replicas spread when nodes are available, and a PDB allowing at most one voluntary disruption. A blank zone means the Omni template has not yet been applied to that machine set; do not assume the Git edit labels a live Talos node by itself.

Failure and rollback

• If a moved policy is SharedResource, confirm the central wave-4 owner has pruned it before allowing a later-wave owner to retry. Stop if two policies target the same workload.
• If recommendations disappear, confirm the target kind/name/namespace and inspect recommender logs. Do not rename the VPA as a first response.
• If resizing threatens scheduler headroom, disable both updater and admission in Git. The recommender continues learning while the change rolls out.
• Roll back a policy move by restoring its previous owner while preserving its namespace/name. Do not delete checkpoints manually.

Sources of truth are the controller values.yaml, the rendered-policy validator validate-vpa-policies.py, and the node labels in cluster-template-prod-v2.yaml.