Joshua García Nieves
Troubleshooting · Postmortems

Where it
broke

Real failures across my network, homelab and Linux machines - what I saw, what I ruled out, and what it actually was.

Joshua García Nieves - Computer Engineering, UAGM Gurabo · Currently focused on Python
Written blameless: including my own mistakes and what I never solved
174Incidents logged
58Crossed to another layer
45Root cause never confirmed
42Still open or abandoned
01 - The pattern

Most problems are fixed where they appear

I kept a log of every problem across eight troubleshooting sessions and tagged each one twice: the layer where I noticed it and the layer where the cause actually was. 110 of 174 were fixed in the same layer they appeared in. The 58 off the diagonal are the ones that take real diagnosis, because the first place you look is the wrong one.

Where I saw it versus where it was

Rows: the layer where the symptom showed up. Columns: the layer of the root cause. Cells on the diagonal are problems fixed where they appeared.

Read it as a pattern, not a statistic. These are 174 entries from my own logs, tagged with a fixed 12-layer vocabulary, before merging duplicates across sessions. The application layer dominates because that is where most of my time goes, not because it fails most.

One trigger showed up in four different layers

Power coming back after an outage left something broken four separate times: the switch stopped handing out DHCP leases, the firewall's WAN did not pick up an address, the shared dual-boot drive came back flagged dirty, and the monitoring looked down. None of the four has a confirmed cause, so I file them as a pattern. It is also why the next item on my bench is an auto-power-on circuit and a recovery sequence, not another service.

02 - Two cases in depth

The first place I looked was wrong

Case A · Network · routing

The console "had no internet", and the firewall was innocent

SymptomAfter I gave the console a fixed address, its connection test failed at the internet step, so the NAT test reported "Failed". It still received a gateway and DNS servers.
What I ruled outA missing NAT rule - that produces a strict NAT type, not a failure. A dead WAN with traffic failing over to the VPN - the gateway page showed the WAN up. A policy-routing rule sending the console into the tunnel - none on that interface, and no floating rules at all.
What the log showedLive firewall logs filtered on the console: only its DNS queries were leaving through the VPN interface. Everything else took the normal path. So the problem was in routing, not in the firewall rules.
CauseWhile setting the reservation I had typed public DNS servers into its fields by mistake. One of them is also the address the firewall pings to check the VPN tunnel is alive. The firewall installs a /32 host route to that address through the tunnel, and longest-prefix match beats the default route. The queries went into the tunnel, where outbound NAT does not cover the home network, and died there. No DNS, so no "internet".
FixEmptied the DNS fields so the console inherits the firewall's own resolver. Open item: move the tunnel's health-check target to an address no client would ever use.
The symptom named DNS and the fix was in DNS, but the cause lived in the routing table. A health-check address is not just a setting: it becomes a route that every client on the network is subject to.
Case B · Storage · boot order

The photo server "deleted" everything, and nothing was lost

SymptomThe private photo server opened as a fresh install, with even the user account gone.
What I ruled outThat a container restart had wiped the database - the photo library was intact on the NAS, and the damage predated the restart by days. That the encrypted dataset had stayed locked after a reboot - its key was loaded and it was mounted. That NFS was failing to cross into a nested dataset - there were no child datasets.
What I foundThe mount point on the application server was an empty local directory created that day, not the NAS share. The systemd automount unit had been in a failed state for two days: Docker fired about six mount requests in five seconds before NFS finished mounting, systemd hit its start limit and stopped trying.
CauseWith the share missing, the database container started anyway and initialized a brand-new empty cluster on the local directory. The app faithfully showed that empty database. The real database and every photo sat untouched on the NAS the whole time.
FixStopped the stack first so nothing else would be written, reset the failed units, mounted the share, and started the stack on the real database. Pending: make Docker require the mount before starting, and move the database off NFS onto local disk.
A failed mount can look exactly like total data loss. The first move is to stop writing, the second is to check what is actually mounted - before restoring anything from backup.
03 - Field log

More incidents, told briefly

A cross-section of the log. Filter by where it happened; each card says how I found the cause and whether it is actually closed.

NetworkResolved

The console that could only join half its games

Symptom
The game console reported NAT Type 3 and matchmaking kept failing.
How I found it
The firewall rewrites the source port of every outbound flow by default. That is endpoint-dependent mapping, which the console classifies as strict NAT.
Outcome
An outbound NAT rule that preserves the source port for that one console only. I caught the mask set to the whole subnet before saving: that would have applied static ports to every device on the network and invited port collisions.
09→09RFC 4787 · pf static-port
NetworkResolved

Phones dropped WiFi after one second

Symptom
Most phones and a handheld console associated and then disconnected within a second, even on a static address. Smart plugs and the air conditioner stayed connected.
How I found it
Ruled out an exhausted DHCP pool (15 active leases) and the VPN tunnel (disabled, still failing). The access point's automatic band steering was failing with modern dual-band clients.
Outcome
Turned off band steering and split the network into one SSID per band. What changed its behavior in the first place is still unknown.
07→03802.11 band steering
NetworkResolved

File-sharing traffic leaving through the internet link

Symptom
The firewall log showed NFS and RPC ports leaving through the WAN - it looked like the storage was exposed.
How I found it
Blocking the ports and resetting states changed nothing. The state table showed the hypervisor as the source, not an outside host. Its backup storage entry still pointed at the NAS address from before I segmented the network; with no route to that old subnet, packets followed the default route out to the internet.
Outcome
Corrected the storage definition in the hypervisor's config file, because the web UI would not let me edit it.
08→06NFSv3 · RPC · default route
NetworkResolved

A kill switch that never blocked anything

Symptom
With the VPN rule ordered one way the server leaked its real address; ordered the other way, it looked fine.
How I found it
The block rule matched the server going to private networks, not to the internet: its destination was missing the invert flag. Internet traffic never matched it and fell through to the general allow rule.
Outcome
Fixed the rule and tested it the only way a kill switch can be tested: took the tunnel down on purpose. The server returned nothing.
09→09first-match · RFC 1918
NetworkResolved

Certificate error on two nested subdomains

Symptom
Two published game-server dashboards showed "connection not secure" in every browser.
How I found it
Ruled out the tunnel's service type, the browser, and the origin being unreachable. A free edge certificate covers one subdomain level; these were two levels deep.
Outcome
Flattened them to one level and both loaded. The cause is inferred from that result - I did not inspect the certificate.
10→10TLS · wildcard certificates
NetworkMitigated

Dynamic DNS that silently never updated

Symptom
The dynamic-DNS entry never showed a current address or an update time.
How I found it
The service log gave it away: the plugin sends the credential in a legacy key header, while a scoped API token needs a Bearer header.
Outcome
Switched to the legacy account key. It works, but that key has far more reach than a scoped token - a trade-off I accepted knowingly and noted as debt.
12→10DDNS · HTTP auth headers
HomelabMitigated

The fans were not coming from the server

Symptom
The main server seemed to be spinning its fans up, and I went looking for the load.
How I found it
Ruled out a ZFS scrub (last runs on known dates), a process I had misread in a screenshot, and video decoding. The noise came from the managed switch sitting next to the desktop, whose fans ramp with heat.
Outcome
Relocating the switch. Lesson: confirm the source of a symptom before explaining it.
02→02
HomelabResolved

A service stuck restarting with a clean exit code

Symptom
The game server's systemd unit flipped between "activating" and "running", and every exit returned code 0.
How I found it
Exit code 0 looked like success, so I read the journal instead. Access was denied on the world file: an earlier manual run as root had created it, and the service runs as its own unprivileged user.
Outcome
Fixed ownership of the data directory and restarted the unit.
05→06systemd · POSIX permissions
HomelabOpen

Cloning a disk onto a smaller one, block by block

Symptom
Migrating the router node from a hard drive to a smaller SSD: the block copy ended with "no space left on device".
How I found it
The source disk was larger than the target. The backup GPT header and the end of the LVM physical volume landed past the end of the new disk. I repaired the partition table by hand, but the volume itself was never shrunk.
Outcome
The SSD was not put into service. Next attempt: shrink the volume first, or copy at the filesystem level instead of the block level.
04→06GPT · LVM2 · dd
Linux desktopResolved

Random freezes and stuck keys

Symptom
Every few minutes the desktop froze for 1-3 seconds, a key stayed pressed and repeated, and the mouse stuck - even while idle watching video.
How I found it
Ruled out ten hypotheses, including keyboard repeat settings, USB hardware, cursor rendering, GPU sync variables, the WiFi daemon and a kernel-level hang. evtest showed a 24-second gap in key events; the kernel log stayed silent. The compositor's own log showed a loop of DRM hotplug events on the GPU, each one forcing a rescan of every connector.
Outcome
Disabled the three phantom connectors. The freezes stopped. Which connector triggered the storm was never identified.
11→05DRM/KMS hotplug · evdev
Linux desktopAbandoned

The expensive detour

Symptom
Before finding the real cause of the freezes, I blamed the GPU driver and tried downgrading to an older branch.
How I found it
The switch left two driver branches half-installed; DKMS could not build for the running kernel, and a failed snapshot rollback closed the way back. Black screen.
Outcome
Full reinstall. In my own words from that session: "we invested a lot chasing the older driver because of an assumption about the freeze that I never validated."
04→05DKMS
Linux desktopResolved

After a full update, the desktop would not start

Symptom
After a full system update the graphical session did not load at all.
How I found it
From a text console, the driver reported an API mismatch. uname -r still showed the old kernel after rebooting: the new kernel image never reached /boot, so the old kernel module was running against the new userspace driver.
Outcome
Put the kernel image in place, regenerated the initramfs and the bootloader config, rebooted.
11→05kernel modules · initramfs
Linux desktopResolved

A shared game drive that kept going read-only

Symptom
The dual-boot drive shared with Windows stopped mounting, then started throwing disk write errors on game downloads.
How I found it
The volume carried Windows' dirty flag, and it kept coming back. Windows Fast Startup does not fully shut down: it hibernates the kernel session, which leaves NTFS volumes marked as in use, so the Linux driver refuses or restricts them. The write errors outlived the remount: permissions, sandboxing and lockfiles all checked clean. Clearing half-written download files from an interrupted session let downloads resume.
Outcome
Fast Startup disabled, plus a shutdown script that forces a full shutdown instead of the hybrid one. The dirty flag stopped coming back.
06→05NTFS · ntfs3 / ntfs-3g · Fast Startup
Linux laptopResolved

Secure Boot keys that enrolled without errors and still did not stick

Symptom
After creating and enrolling my own Secure Boot keys with no errors, the firmware still reported Setup Mode.
How I found it
Checked whether the tool was showing stale data: a second tool agreed, and reading the raw EFI variable returned 01. The firmware accepted the KEK and db keys from the OS but refused to persist the Platform Key through that path.
Outcome
Built signed .auth files, copied them to the EFI partition, rebooted into the firmware and enrolled the Platform Key from its own menu. Setup Mode: disabled.
03→03UEFI Secure Boot · PK/KEK/db
Linux laptopMitigated

A boot partition too small, and a failure that cascaded

Symptom
The bootloader hook kept failing to copy the initramfs, and then the regenerated boot menu came back with no kernels in it.
How I found it
The EFI partition was 256 MB and each initramfs about 245 MB; the tool copies kernels into that partition. When the copy failed, the tool aborted before writing any boot entries - so one space problem looked like two bugs.
Outcome
Wrote the bootloader config by hand, pointing at a separate 1 GB /boot partition instead of duplicating kernels into the EFI partition.
06→06UEFI ESP · FAT32
Linux laptopResolved

Login loop right after installing

Symptom
At the login screen the password was accepted, the screen flickered and came back to the login.
How I found it
The display manager's journal showed a successful login for root followed by an immediate logout. The installer script wrote the autologin user from $USER, and because it ran under sudo, $USER expanded to root.
Outcome
Corrected the user in the autologin config. The installer bug is worth reporting upstream.
12→05sudo · environment variables
04 - Still open

A laptop that shuts off when you plug it in

Electronics · power delivery
SymptomPlugging in the 240 W charger shuts the laptop off instantly - while running, while booting, even sitting in the firmware setup screen. Over two weeks it progressed: it stopped booting on battery alone, then started shutting off on USB-C power too.
What I ruled outThe charger - a new 240 W unit from a different brand fails the same way. A fault limited to the barrel input - USB-C now fails too. The operating system - it happens in the firmware screen, with no OS loaded.
What I knowResetting the embedded controller (battery disconnected, power held, reconnected) restored booting once, inconsistently. The battery reports the model string of a much older laptop while its design capacity matches this one. There is no visible heat damage around the power input.
Two hypotheses still standingA degrading component on the power path shared by the barrel and USB-C inputs, possibly made worse by repeated failed power-ons; or an embedded-controller firmware problem negotiating power - which on its own does not explain the USB-C failure.