Joshua García Nieves
Homelab · Puerto Rico

Megaman
Homelab

Production home infrastructure, run with engineering discipline: measure before acting, document afterwards.

2Hypervisors
4Segmented VLANs
28 TBRaw across 3 ZFS pools
~22Services in production

01 - The system

A lab that isn't allowed to go down

This homelab has one constraint that changes everything: my family depends on it. It holds the household's photos, the media server, the security cameras and the internet routing. There is no maintenance window and no "it's just a lab" excuse. Every architectural decision has to survive a 3 AM power outage with nobody awake to babysit it.

That constraint is exactly what makes it a good place to practice. I learn networking, storage and virtualization on a system where the consequences are real, and that forces me to reason in terms of failure domains instead of just stacking up services.

Architecture - 802.1Q segmentation and failure-domain separation
Internet (ISP) Managed switch 802.1Q switching in hardware · per-port tagging Ports assigned by VLAN · management restricted to its own segment trunk link - all four VLANs tagged (router-on-a-stick) Network node dedicated hypervisor · separate hardware Virtualized firewall / router inter-VLAN rules · DHCP · NAT · policy-routed WireGuard tunnel VLAN · WAN Carrier uplink, isolated from the rest of the internal network VLAN · Servers Hypervisors, NAS and services. AI server via VPN + kill switch VLAN · Home Personal devices, WiFi and console. Direct egress VLAN · Cameras No internet. RTSP to the video server and nothing else Compute and storage node hypervisor separate from routing Virtualized NAS 3 ZFS pools: double parity, mirror and stripe. Exported over NFS to the apps Application node ~22 containers. Dedicated GPU over PCIe for vision and local models

The architectural decision I defend most: the virtualized router runs on a different hypervisor from compute and storage. If it lived on the same box, shutting down the file server would take the house offline, and rebooting the network would leave storage without clients. Separating failure domains costs one extra machine and removes an entire circular dependency.


02 - Four real problems

What I learned solving them

Case 01 · Data analysis

An absolute number misled me for months

ProblemMy disks reported very different counts of reallocated sectors. For months I prioritized replacement by the highest number, which seemed obvious.
What I measuredI pulled not just the defect count from each disk but also its power-on hours, and divided one by the other to get a rate: defects per 1,000 hours of life.
Rate of decay, not accumulated count

Defects per 1,000 power-on hours. Each disk's absolute count sits under its name - the chart's order does not match the order of those numbers.

Primary pool (double parity)Striped pool (no redundancy)Mirrored pool
defects per 1,000 power-on hours - scale 0 to 35
What I foundThe disk with 190 defects is degrading faster than the one with 1,947. The second took seven years to get there; the first has been running ten months. And the fast one sits in the production pool - the one holding the family's photos - not in the accept-the-risk pool where I thought my problem was.
What I ruled outI checked whether the cause was environmental: temperature, power cycles, age. The hottest disk in the fleet has zero defects, and the two with the most power cycles are the healthiest. Four units from the same batch, same hours, same chassis: two flawless and two degrading. That's unit variation, not environment - so the answer is replace the drive, not redesign the cooling.
Case 02 · Operational risk

If a disk failed, I didn't know which one to pull

ProblemThe filesystem identified its members by internal identifiers, and the OS by letters the kernel reassigns on every boot. Neither pointed at a specific physical drive. On top of that, a motherboard limitation keeps the storage controller out of passthrough, so the NAS sees identifiers invented by the hypervisor rather than the manufacturer's.
Why it mattersOne of the pools has no redundancy. Opening the wrong bay there isn't a scare - it's losing the whole pool. And I found that two different configurations of the same NAS mapped the same disks to different slots, so the slot number alone was a false lead.
What I didI rebuilt the full identity chain for all eight disks: filesystem identifier → virtual slot → manufacturer's world-wide name → serial number → physical position. I froze it in a versioned document and adopted the rule of never operating on a device letter.
An identifier that changes between reboots identifies nothing. Before automating or intervening, make sure the name you're using always points at the same physical object.
Case 03 · Method

I tested my own hypothesis and I was wrong

SuspicionI believed the UPS shutdown system had a race condition - that the battery cut power before the server finished shutting down, abusing the disks on every outage.
How I tested itInstead of "fixing" it, I measured it. I pulled the full sequence of three real outages from the system logs, with timestamps. The virtual machines shut down first, one after another, each with a clean close, and the host follows. The whole sequence takes about 50 seconds against the 180 the configured threshold guarantees.
ResultThe hypothesis was false. And when I cross-checked disk degradation against power-cycle counts, that correlation didn't appear either: the disks with the most outages are the healthiest ones.
The second timeShortly after, I concluded the UPS battery was degrading, because runtime had dropped from 22 minutes to 14. I was wrong again: I had plugged my workstation into the same UPS. More load, less runtime - and with lead-acid the drop isn't even proportional. The battery was fine. Since then I never record a runtime figure without recording the load next to it.
A reasonable suspicion is not a diagnosis. I'd rather kill my own theory with data than change a setting that may never have been wrong - measuring costs far less than fixing what wasn't broken.
Case 04 · Network security

I isolated what I can't audit

ProblemIP cameras are the least trustworthy device on any home network: closed firmware, sparse updates, and an attack surface I can't review.
DesignI put them on their own VLAN with 802.1Q tagging and wrote explicit rules: they may speak only RTSP to the video server, and nothing else. No internet egress, no access to the home network. Everything else is denied by default.
What it cost me to learnRules are evaluated first-match, top-down, and I learned it on the rule next door. The kill switch for my AI server sat in the wrong position and was missing its inverted destination, so it matched private networks instead of the internet and never blocked anything. The firewall's live log showed which rule each packet actually hit. Then I tested it the only way a kill switch can be tested: take the tunnel down on purpose and confirm the server gets no answer at all.
ExtensionI applied the same criterion to egress: the AI server leaves through a policy-routed VPN tunnel, and if the tunnel drops it goes dark instead of falling back to the ISP. Home traffic exits directly. Segment by trust, not by convenience.
Segmentation is damage containment, not paranoia. The question isn't whether a device will fail, but what it can reach when it does.

03 - Still open

I know what's missing

The two open fronts are automating virtual-machine backups and migrating a pool that has no redundancy. Both are documented with their prioritization criteria and a plan. Knowing what's incomplete and why it isn't yet the priority is part of the job.