Linux Worked Example — The Server That Nobody Could Reboot
A worked example of a critical host built by hand five years earlier — how its configuration was recovered, what the rebuild found, and why the fear of rebooting it was justified.
This is an illustrative example. The company, host and figures are invented. The situation — one essential server that everybody works around and nobody will touch — exists in most organisations over five years old.
The situation#
A manufacturing company. One server, built in 2021 by a contractor, ran the integration layer between the production floor systems and the ERP. Everything went through it.
Nobody had rebooted it since 2023. The reasons had hardened into folklore:
- Nobody knew everything running on it
- It had been built by hand and there was no build script
- The last unplanned reboot took six hours to recover from
- The contractor had left
It had 1,100 days of uptime, which several people mentioned with pride. It also had 61 pending security updates including four kernel updates, none of which could apply without the reboot nobody would authorise.
What forced the issue#
The disk reached 94%. Growth said full within three weeks. A full root filesystem would take the host down without warning, at a moment nobody chose — which is the same outage everyone was avoiding, arriving on its own terms instead.
Recovering what it did#
Three days, and the method matters more than the result.
Everything listening. ss -tulpn, then trace each port to a process, each process to a unit or an init script, each of those to a package or a directory.
| Found | |
|---|---|
| Listening ports | 14 |
| Explained by documentation | 3 |
| Explained by asking people | 6 |
| Traced by reading configuration | 4 |
| Never explained | 1 |
Everything scheduled. System and per-user crontabs, timers, and one script that re-added itself to cron on each run — added by someone who had been fighting a colleague's cleanup script in 2022.
| Found | |
|---|---|
| Scheduled jobs | 23 |
| Still doing something useful | 14 |
| Failing silently for over a year | 6 |
| Writing to a filesystem that no longer existed | 3 |
Everything on disk that was not from a package. This is where the 94% came from.
| Size | |
|---|---|
| Application logs, never rotated since 2021 | 180 GB |
| Database backups written locally, never removed | 71 GB |
| A core dump from 2022 | 14 GB |
| Copies of the ERP export, one per day, kept forever | 43 GB |
| The application itself | 2 GB |
Rotation had been configured. It had never worked, because the configuration named a path that changed in 2021, and nothing checks whether a rotation configuration actually rotates.
The unexplained port#
The one port nobody could account for was a small Python process, started from a systemd unit written in 2021, listening on all interfaces, serving a JSON status endpoint with no authentication.
The endpoint disclosed production line names, current job identifiers and the ERP hostname. It was reachable from the office network. It had been running for four years.
It had been built by the contractor as a debugging aid.
The rebuild#
Two weeks, in parallel with the old host still running.
Configuration management from the start. Every finding became a line in the definition rather than a step in a document. The rule adopted: if it is not in the definition it does not exist on the new host, which forced a decision on every one of the 23 jobs and 14 ports rather than carrying them across by default.
Six failing jobs deleted. Investigated first, and none of them had been noticed as absent — the work they had done was either no longer needed or had been quietly taken over elsewhere.
Log rotation configured, and verified by fast-forwarding time in a test instance until rotated files appeared. Configuring it was the mistake that had been made in 2021. Verifying it was the fix.
Backups moved off the host. Local backups on the machine being backed up had been the second largest consumer of disk and would have been worthless in the failure they existed for.
The unexplained service was not carried over. Nothing broke.
The cutover#
Ninety minutes on a Saturday. Traffic switched, watched for two hours, old host left powered on for a fortnight and then destroyed.
The rebuild that everyone had feared for two years took two weeks of one person's time, and the fear was entirely reasonable — the host really was undocumented and really did have surprises on it. What was not reasonable was that the fear had prevented the work rather than prompted it.
The result#
| Old host | New | |
|---|---|---|
| Build method | By hand, 2021 | Configuration management |
| Time to rebuild from scratch | Unknown | 25 min, tested |
| Disk usage | 94% | 31%, with rotation verified |
| Pending security updates | 61 | 0 |
| Unexplained listening ports | 1 | 0 |
| Scheduled jobs failing silently | 6 | 0 |
| Backups | Local, untested | Off-host, restore tested |
| Uptime | 1,100 days | 12 days, and rebooted twice on purpose |
The last row is the point. Twelve days of uptime on a host that can be rebuilt in 25 minutes is a stronger position than 1,100 days on one that cannot.
What was learned#
High uptime on a host you cannot rebuild is a liability presented as an achievement. It measures how long it has been since anyone dared to touch it.
Configured is not working. Log rotation was configured for five years and rotated nothing. The checklist question has to be "did it rotate?", not "is it configured?".
Rebuild rather than clean up. Cleaning the old host would have recovered the disk and left every other finding in place, including a service disclosing production data to the office network.
Make the definition the authority. Requiring every carried-over item to be justified is what removed six dead jobs, one unauthenticated service and 43 GB of exports nobody read.