Your homelab:
what to fix, keep, and try.

Proposed procedure

Prerequisites: access to Tailscale DNS/network administration; a saved copy of existing DNS settings; a phone on cellular or another genuinely offsite client. Recheck the TrueNAS subnet route and intended client permissions first.

1. Remove the stale global 192.168.1.1 nameserver. Choose either reachable public global resolvers or clients' local DNS; review the global override accordingly.
2. Add 192.168.10.15 as a restricted nameserver for vanillax.me.
3. Keep the NAS's approved 192.168.10.0/24 route. Ensure intended clients can reach the DNS server on TCP/UDP 53 and the app gateway on HTTPS.
4. Enable “Use Tailscale DNS” on the test client. Linux clients also need accepted subnet routes. Review “Use with exit node” if the client uses one.
5. On cellular, check private resolution, unrelated public DNS, and an actual HTTPS app.

Success looks like this

• argocd.vanillax.me resolves to 192.168.10.52.
• news.vanillax.me resolves publicly, and google.com also resolves.
• The private app opens over HTTPS. Correct DNS alone does not prove application routing works.

On a connected Mac, check the system resolver with this read-only command:

dscacheutil -q host -a name argocd.vanillax.me

If validation fails: separate DNS resolution from subnet reachability. If names resolve but HTTPS fails, inspect client route acceptance and access policy. Restore the saved DNS settings to undo the change, understanding that the old global resolver still depends on the offline router.

Do not blindly toggle NAS settings: its accept-dns=false and accept-routes=false govern the NAS itself. They do not disable its advertised subnet routing for other clients.

Confidence boundary: the configuration fault is verified. All other tailnet peers were offline during inspection, so the user's offsite symptom and proposed remediation were not tested end to end. Tailscale DNS reference · Subnet-router documentation.

The HGST He10 specification gives about 249 MB/s sustained per drive and mechanical seek/rotation delays; the Seagate 10 TB reference is up to 254 MB/s. A mirrored ZFS pool is not guaranteed the sum of all its drive specifications. Samsung and HP SATA SSD ratings are around 550 MB/s per drive under vendor conditions. This is a specification comparison, not an industry percentile.

BigTank’s approximately 10.2 ms average uncached random read is in mechanical-latency territory. Its 108.5 ms read p99 matters more to an interactive VM than the bulk MB/s headline. The SSD pool’s 396 MB/s flushed write result warrants a longer representative test if writes become the priority; it does not diagnose a bad drive.

A 10 GbE link has a 1.25 GB/s arithmetic ceiling before overhead; 2.5 GbE has 312.5 MB/s. The historical 1,119 MB/s TCP test equals 8.95 Gb/s. The SSD pool can approach the network budget; cached RAM reads exceed it.

Canonical NAS reference, drive datasheets and methodology · Raw measurement summary.

Two observation windows

72-hour history: 432 ten-minute averages from existing Netdata metrics. CPU mean 1.51%, p95 2.43%, maximum ten-minute average 11.88%. These averages can hide brief spikes. Demand-data hit rate was calculated from access/hit rates because the supplied percentage chart returned invalid zeros.

Five-minute live sample: 20 September, 17:16:08–17:21:08 UTC. Demand-data reads: 405,024 hits and 15 misses, or 99.9963%. CPU averaged 3.68%; the highest 15-second sample was 10.98%. Traffic averaged 8.51 MiB/s inbound and 6.56 MiB/s outbound. The highest observed HDD utilization over a 15-second sample was 27.5%.

There was no observed CPU, disk, or memory saturation during that workload. That does not test the NAS at maximum load.

Why the smaller-cache history is only a clue

About 11 hours of historical buckets had ARC below 128 GiB and roughly 99.999967% demand-data hits. That suggests some workloads fit comfortably in less cache. Those periods had different activity and cache warm-up conditions, however, and do not simulate a machine with only 128 GB total RAM.

Benchmark controls and limitations

The complete NAS suite ran 17:45:56–17:54:58 UTC: 24 tests on three new 8 GiB-quota scratch datasets. Disk phases used primarycache=metadata, secondarycache=none; warm phases used primarycache=all after a warmup read. Metadata caching remained enabled. Prefetch can generate ARC demand hits while reading from disk, so physical disk-byte deltas establish the data path.

Sequential writes used buffered I/O plus end_fsync=1; random writes used fsync=1. The latter’s full write cycle includes both the write and separate fsync. All temporary datasets were removed, production properties were unchanged, and the pools stayed healthy. No global cache drops or ARC-limit changes were performed.

The first small read-only pass is preserved as historical evidence in the NAS reference. Its 96.7 MB/s old-file direct read is not the pool maximum. An old 4 GiB sparse file with only 512 bytes allocated was excluded. Historical NFS/SMB tests used different sync settings and do not establish protocol overhead.

The next experiment that would answer the RAM question

1. Choose an acceptable workload target: backup completion window, restore duration, Steam behavior, and file-open latency.
2. Record a full representative workload cycle with current RAM: app memory peaks, demand-data hit/miss rates, disk latency/queues, and task duration.
3. In a planned experiment, compare smaller ARC budgets while reserving enough memory for the OS and application peaks. Changing ARC size alone is not a perfect simulation of smaller physical RAM.
4. Repeat comparable workloads after a defined warm-up period. Keep a configuration record and restore the original ARC settings if latency, memory pressure, or task duration worsens beyond the chosen target.
5. Use the observed difference and hardware price to choose between 128, 256, and the current capacity.

This is a proposed experiment, not a change performed during the audit. ARC interpretation: OpenZFS caching and ARC statistics.

All ten disks passed SMART, and pools reported no errors. The day's Backup10T scrub completed without errors. SMART success does not guarantee future reliability.

The HP S700 reported 816 lifetime CRC errors; the Samsung 860 EVO reported four. No matching recent kernel I/O errors were found. Save the counters and compare later before treating historical counts as an active cable or controller fault.

Do not compare SMART internal temperatures directly with a manufacturer's ambient-temperature limit. This assessment recommends improved cooling based on the observed temperatures; it does not claim the drives exceeded a verified manufacturer threshold.

Available host memory is reclaimable memory at observation time, not unused guest allocation. The recently booted shed VM had not touched its full allocation. All active Talos VMs have ballooning disabled. Host samples are not peak-load guarantees.

These are host OS disks, not interchangeable VM disk benchmarks. SFF’s worker shares the tested physical disk, but its control-plane disk is separate. Elite’s VM uses a different Intel SSD. The Threadripper test is not its enterprise mirror. Physical counters confirmed roughly 1 GiB read on each tested root disk.

Control plane: memory p95 5.91 GiB, sampled maximum 6.16 GiB. History ends at 17:40 UTC, before our benchmarks; five-minute rates were sampled every ten minutes and can miss spikes. Shed coverage is incomplete. Kubernetes schedules against requests, so instantaneous free memory is not the entire placement budget.

Revised guest tests used one Python process, a new 512 MiB file, 1 MiB direct-I/O sequential writes/reads, and 128 separate 8 KiB writes with fsync. Every direct-write phase included a final fsync. Guest/client page cache was bypassed for sequential I/O; upstream caches and production traffic remained.

The NAS SMB dataset has sync=disabled, verified live. Its fast fsync replies do not establish durable storage and should not be compared as equal guarantees with the Longhorn paths. The read can come from NAS ARC.

The GPU enterprise path saw about 8.1 GB of guest-device reads while the benchmark itself read about 0.54 GB. Its 46.8 ms write-cycle p99 reflects a busy application path; it does not establish that enterprise drives are intrinsically slower. These tests cross different layers, so subtracting a host result from a guest result would not measure “Longhorn overhead.”

The seven revised tests completed without reported benchmark errors. No raw VM disk, production file, resource limit, or scheduling configuration was changed. Small samples and current workload competition limit generalization; no synthetic storage test establishes a supported web-user or database-user count.

Full fleet inventory, exact methods and raw evidence.

Hot: NAS HDDs at 54–58°C justify checking cooling. Threadripper’s 103.6°C sensor was Tctl, not the die temperature; Tdie peaked at 76.6°C in the historical query. Interpret sensor labels before diagnosing overheating. Linux AMD temperature-sensor definitions.

Slow: the shed’s remote path, SFF contention, and some storage tail latency deserve attention. Etcd’s WAL p99 exceeded the upstream 10 ms guideline in about 9.14% of sampled windows; backend commit p99 exceeded 25 ms in about 0.99%. Etcd performance guidance.

Underused: NAS CPU and some guest memory have headroom in observed activity. That does not mean unused hosts can be switched off: 44 volumes have their only copy on GPU, the shed is tainted against ordinary scheduling, Elite has attached devices, and SFF owns the sole control plane. Four pretest OOM counter events on each of GPU and Elite remain unattributed; do not equate them automatically with host RAM exhaustion.

Next measurable gate: choose one ordinary Dell app, establish a recoverable data copy and an eligible destination, then plan its movement through GitOps. Keep rollback storage and the old host until the app and its backup/restore behavior are verified. No migration was attempted by this assessment.

Four hosts use hostname pve. Every host has its own VM 100. Running Talos guests use cpu: host, and the fleet mixes AMD and Intel processors. Per-host storage names and local virtual disks also differ.

Before joining a cluster, you would need unique node identities, a plan for guest evacuation/backup and restore, noncolliding VM IDs, and reconciliation of storage and Omni provider configuration. PVE cluster join overwrites local cluster configuration and is not an ordinary dashboard setting. The previous Corosync incidents were not forensically diagnosed in this investigation.

What the Pi is doing

The Pi runs Ubuntu on ARM64 with about 7.7 GiB RAM. ProxCenter's frontend and PostgreSQL used approximately 140 MiB and 27 MiB respectively in one snapshot. Around 6.1 GiB was available overall. Omni, its five providers, and Technitium also run there. There is no measured resource reason to move the dashboard.

Evaluate the PDM installation you already have

PDM 1.1.7 is installed and both API services are active in Elite LXC 101. Its HTTPS endpoint returned HTTP 200; five remote definitions point to your five PVE hosts. This confirms a working service and configuration, not successful authentication or every remote operation.

Allow 30–60 minutes to compare inventory freshness, console access, backup visibility and a harmless operation on a disposable guest in PDM and ProxCenter. Keep the tool that supports your actual workflow. PDM downtime does not stop managed guests, but placing it on Elite makes its availability depend on that host.

No new PDM installation or removal is needed for this comparison. Save/export relevant configuration before changing remote definitions; revoke unused credentials only after choosing and validating the retained tool. PDM capabilities.

When a real cluster becomes worth it

Revisit a three-node, reliably wired design when you have a specific guest-failover requirement, a compatible storage plan, and enough spare capacity for failures. Keep the shed out of that quorum proposal. Proxmox calls for stable sub-5 ms networking and recommends separating Corosync from heavy storage traffic. HA also requires addressing passthrough, CPU compatibility, and guest storage; joining hosts alone does not solve them.

Can the control plane run in Kubernetes?

Yes, technically. Modern self-hosted deployments combine management, signal, relay, and STUN in netbird-server, with a dashboard and embedded identity support. The upstream quickstart starts at one CPU and 2 GB RAM; that is a provisioning baseline, not a measurement of this lab.

HTTP, WebSockets, OAuth, and gRPC/h2c must route correctly. Public UDP 3478 for STUN needs direct reachability; a Cloudflare HTTP Tunnel cannot supply it. Home hosting therefore needs appropriate inbound reachability, or an external endpoint such as a VPS. Quickstart · Proxy and transport requirements.

Inside this repository, an implementation would need pinned manifests, named Services, explicit network permissions, 1Password-managed secrets, backed-up state, and an update strategy appropriate for a single-writer database/PVC. Kubernetes, Cilium, secrets, and storage then become dependencies of remote access.

Two integration details to verify

• The official server Helm chart source inspected in the investigation still described the older split-service architecture (chart 1.9.0 / app 0.46.0). Verify it against the intended current release before using it. The Kubernetes operator attaches services to an existing management account; it is not the control-plane server.
• NetBird Gateway API integration exists, but upstream labels it beta. Its self-hosted public-proxy documentation identifies Traefik as the supported external TLS-passthrough edge. This is not a verified drop-in substitution for this lab's Cilium gateway.

Server chart source · Kubernetes operator · Gateway API status.

One possible public path

This adds VPS cost, patching, backup, and bandwidth responsibilities. No VPS was selected or purchased. Direct peer paths may be fast; relayed paths depend on relay bandwidth and latency. Remote NAS traffic also depends on your WAN uplink.

Acceptance and rollback

1. Pick private VPN access or public ingress as the first goal.
2. Test one client and one noncritical resource while the existing path remains available.
3. Verify DNS, intended access policy, reconnect behavior, direct versus relayed paths, and usable throughput.
4. Test recovery access during an isolated pilot-component outage; do not take down the production cluster to prove it.
5. Only replace an existing endpoint after behavior meets your needs. Roll back by returning clients/DNS to the original path and removing pilot routes and credentials.

Switching VPN products would not automatically repair Technitium routing or the current DNS policy. Those dependencies must be configured in either product. This investigation did not verify that Cloudflare Access authentication is active; do not assume a tunnel alone supplies it.

Broad prerequisites match; exact compatibility remains untested

All hosts run PVE 9.2.18. The NAS runs TrueNAS-26.0.0-MASTER+20260914-020141. Upstream's minimum is TrueNAS 25.10+ and PVE 8+ for iSCSI, or PVE 9+ for NVMe/TCP. The development NAS version meets that broad floor but requires testing against the chosen plugin release.

On .14, open-iscsi and nvme-cli were present and the plugin was absent. NAS iSCSI and NVMe-oF services were running. PVE's live storage API version was 15 with compatibility age 6; an older plugin API declaration alone is not proof of incompatibility.

Choose matching release artifacts

On 20 September, v2.1.23-beta4 was the newest published release (19 September; marked prerelease). GitHub's “latest” endpoint selected beta3, whose prerelease flag was false despite its beta name. The main-branch version banner was older still. Pin a coherent package/tag and its matching configuration documentation; do not combine moving-branch installer files and unrelated release instructions.

The Reddit discussion mentions a dataset-creation bug and a beta-tag installer error. Tagged source shows several historical defects already fixed, including the PR82 dataset issue, later cloud-init changes, and beta4 capacity-cache/FQDN fixes. Open issue titles alone do not establish that the chosen release still fails. No plugin runtime defects were reproduced here.

Beta4 release · Tagged changelog · Cleanup/VMID-reuse report to validate.

Credentials and isolation

Use a dedicated API identity, verify the documented roles against the installed TrueNAS version, and keep credentials in 1Password and private host storage. Keep TLS verification enabled with a trusted matching certificate. Dataset write/delete roles may grant appliance-wide capabilities: selecting a pilot dataset prevents naming collisions but does not necessarily restrict the API key to that dataset. Restrict storage initiators as appropriate. API permissions guide.

Keep the current Talos storage design during the trial

Most active Talos disks are on per-host local LVM-backed storage. Moving their Longhorn storage disks onto this one NAS would place those volumes—and any replicas hosted there—behind the same physical NAS dependency. The plugin does not remove GPU/USB passthrough or mixed-CPU migration constraints.

For important future VM data, use a redundant pool. The current ai-pool is intentionally striped without redundancy. Snapshot support does not replace independent backups. Adding VM workloads would also change the NAS working set, so revisit RAM sizing after that change rather than applying today's measurements unchanged.

Stop conditions and rollback

Stop the pilot on unexplained data-integrity errors, failed cleanup, or a failed move back to local storage. Preserve logs and the exact version; do not test deletion on production volumes. Restore the disposable guest from its known-good backup or local copy. After ensuring no guest uses the pilot backend, remove only the pilot storage configuration, targets, data, and credentials. Do not stop shared NAS services to remove the trial.

Any future repository-owned implementation follows the branch/PR workflow. Host installation and storage changes need a concrete deployment plan; this report is the assessment, not that deployment.