Network Topology

Overview

The cluster uses a single network with 10G switch infrastructure:

Main LAN (192.168.10.0/24) - All cluster traffic via 10G switch
TrueNAS Storage - 192.168.10.133 (10G connected via switch)

Physical Topology

┌─────────────────────────────────────────────────────────────────────────────┐
│                              NETWORK TOPOLOGY                                │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│   ┌─────────────────┐                                ┌─────────────────┐    │
│   │    Proxmox      │                                │    TrueNAS      │    │
│   │   hp-server-1   │                                │  192.168.10.133 │    │
│   │  192.168.10.14  │                                │                 │    │
│   └────────┬────────┘                                └────────┬────────┘    │
│            │ 10G                                              │ 10G         │
│            │                                                  │             │
│            ▼                                                  ▼             │
│   ┌────────────────────────────────────────────────────────────────────┐   │
│   │                        10G SWITCH                                   │   │
│   │                     192.168.10.0/24                                 │   │
│   └────────────────────────────────────────────────────────────────────┘   │
│            │              │              │              │                    │
│            ▼              ▼              ▼              ▼                    │
│   ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐       │
│   │ Control Plane│ │ Control Plane│ │ Control Plane│ │   Workers    │       │
│   │  .237        │ │  .76         │ │  .140        │ │ .164/.219/.159│      │
│   └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘       │
│                                                                              │
│   ┌──────────────────────────────────────────────────────────────────┐      │
│   │                        GPU Worker VM 100                          │      │
│   │  ┌─────────────────┐                                             │      │
│   │  │ net0 (ens18)    │                                             │      │
│   │  │ vmbr0 → 10G LAN │                                             │      │
│   │  │ 192.168.10.x    │                                             │      │
│   │  │ (DHCP)          │                                             │      │
│   │  └─────────────────┘                                             │      │
│   └──────────────────────────────────────────────────────────────────┘      │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

IP Assignments

Main LAN (192.168.10.0/24)

Device	IP	Purpose
Router/Gateway	192.168.10.1	Default route
Proxmox (hp-server-1)	192.168.10.14	Hypervisor
TrueNAS	192.168.10.133	NAS (NFS/SMB/RustFS S3) - 10G
Control Plane 1	192.168.10.237	K8s master
Control Plane 2	192.168.10.76	K8s master
Control Plane 3	192.168.10.140	K8s master
Worker 1	192.168.10.164	K8s worker
Worker 2	192.168.10.219	K8s worker
Worker 3	192.168.10.159	K8s worker
GPU Worker	192.168.10.x (DHCP)	K8s GPU worker
Wyze Bridge	192.168.10.46	RTSP camera streams
LoadBalancer Pool	192.168.10.32-63 (/27)	Cilium L2 announcements

Talos Configuration

machine:
  network:
    interfaces:
      - interface: ens18
        dhcp: true
  kubelet:
    nodeIP:
      validSubnets:
        - 192.168.10.0/24

Proxmox Bridge Configuration

Bridge	Physical NIC	CIDR	Purpose
vmbr0	ens2	192.168.10.14/24	Main LAN (10G)

TrueNAS Network Configuration

Interface	IP	Speed	Purpose
enp67s0	192.168.10.133/24	10G SFP+	Main LAN (via 10G switch)

Whitelisted Storage Access

The Cilium network policy allows these storage connections:

Destination	Ports	Purpose
192.168.10.133	2049, 111	NFS
192.168.10.133	445	SMB
192.168.10.133	9000, 30292, 30293	RustFS S3 (Loki, Tempo, pgBackRest)

Troubleshooting

Can’t Reach Storage

# Test connectivity to TrueNAS
ping 192.168.10.133

# Test NFS mount
showmount -e 192.168.10.133

Storage Performance Testing

# Test raw wire speed (should be ~9.4 Gbps)
iperf3 -c 192.168.10.133

# Test NFS throughput from inside a pod
kubectl exec -n <ns> <pod> -- dd if=/mnt/nfs/testfile of=/dev/null bs=1M status=progress

# Test NFS throughput from Proxmox host (bypasses VM layer)
mount -t nfs -o nfsvers=4.1,nconnect=16,rsize=1048576,wsize=1048576 192.168.10.133:/mnt/BigTank/k8s/llama-cpp /mnt/nfstest
dd if=/mnt/nfstest/testfile of=/dev/null bs=1M status=progress

NFS 10G Tuning

The default Linux kernel read_ahead_kb of 128 KB limits NFS sequential reads to ~140 MB/s on any link speed. The cluster applies these fixes via Talos machine config:

Layer	Setting	Value
VFS readahead	udev rule `ATTR{read_ahead_kb}`	16384 (16MB)
NFS readahead	`siderolabs/nfsrahead` extension	Installed on all nodes
RPC concurrency	`sunrpc.tcp_slot_table_entries`	128 (default was 2)
TCP congestion	`net.ipv4.tcp_congestion_control`	bbr
TCP buffers	`net.core.rmem_max` / `wmem_max`	64MB
NIC ring buffers	Proxmox + TrueNAS	8192 (max)
NFS mount options	Per-PV CSI mountOptions	`nconnect=16,rsize=1M,wsize=1M`

Verified performance (from TrueNAS ARC-cached 4GB file):

Layer	Speed
iperf3 (wire)	9.4 Gb/s
Proxmox host → NFS	2.7 GB/s
Talos VM → NFS (before tuning)	~128 MB/s

Debug commands:

# Verify readahead is 16384 (not 128)
kubectl exec -n <ns> <pod> -- cat /sys/class/bdi/0:*/read_ahead_kb

# Verify sunrpc slots are 128 (not 2)
kubectl exec -n <ns> <pod> -- cat /proc/sys/sunrpc/tcp_slot_table_entries

# Full NFS mount stats (connections, slots, RTT)
kubectl exec -n <ns> <pod> -- cat /proc/self/mountstats

See scripts/debug-nfs-server.sh (TrueNAS) and scripts/debug-nfs-client.sh (Proxmox) for comprehensive debugging.