Linux Troubleshooting Diagram
A path through a server problem that does not depend on guessing — the four resources to check in order, what each one's symptom looks like, and why the obvious suspect is usually not the cause.
SVG. No sign-up, no email.
Most server troubleshooting is guessing dressed as intuition: someone remembers a previous incident and checks that first. The path below is deliberately ordered instead — cheapest and most-often-true first — and it resolves the majority of problems before anything clever is needed.
The first lane resolves most incidents#
Disk full is the single most common cause of "the server is broken". It presents as almost anything: a service that will not start, a database refusing writes, logins that hang, a web server returning errors it has never returned before. Check inodes as well as bytes — a partition with free space and no free inodes fails in exactly the same way and shows plenty of room in the obvious command.
Out of memory is second, and it is deceptive because the evidence is not where the symptom is. The kernel's OOM killer selects a process, terminates it, and logs the decision. The service that disappeared is often not the one using the memory — it was the one that looked most expendable when something else exhausted the machine. Read the log rather than reasoning about which process seems suspicious.
Time is last in the lane and worth keeping there. Clock drift produces failures that look like nothing to do with time: TLS handshakes rejected, tokens considered expired, authentication refused, log timestamps that make the timeline unreadable during the incident you are trying to solve.
The second lane is where discipline pays#
"What changed?" is the highest-value question in operations and it is skipped constantly, because investigating feels more like work than reading a change log. A deploy, a patch, a configuration push, a certificate rotation, a dependency upgrade — one of these precedes most incidents, and the change log answers in a minute what an investigation answers in an hour.
Testing from the host itself splits the problem in half. If the service responds locally and not remotely, the cause is between the two: firewall, load balancer, DNS, routing, security group. If it fails locally too, none of those matter. This one check eliminates most of the possibility space and it takes seconds.
One host or all of them does the same for the other axis. A problem on one host is the host. A problem on every host is the service, its dependency, or something shared.
The third lane, and why it is drawn as a set#
These are not sequential — they are the less common causes, checked once the ordinary ones are excluded.
File descriptor limits appear under load and look like a network problem. The service stops accepting connections while the machine is healthy by every other measure, because the limit is per-process and defaults are often low for a server doing real work.
DNS is worth checking for slowness, not just failure. A resolver that is timing out and falling back to a second entry adds seconds to every lookup, which appears as a service that became mysteriously slow with no change to it.
Certificate expiry is drawn muted because it is rarely the answer, and it belongs on the list because when it is the answer it is the answer completely, and it wastes an hour of network investigation first.
What this diagram is really for#
It is an ordering, and the ordering is the value. Any competent engineer can check all of these. Under pressure at 3am, the difference between resolving an incident in ten minutes and in ninety is whether the checks happen in the order of how often each one is true, or in the order they come to mind.
Print it, put it in the runbook, and follow it even when you are confident you know the cause — particularly then.