The Safety Report That Never Arrived: A Fictional SysOps Story

Nothing here happened to a real school: the week described is made up. The technical details, though, all rest on the Ops Atlas page about SysOps, the craft of keeping servers healthy. Picture a small flight academy whose safety management system lives on one Linux server. The academy has a safety manager, a part-time IT contractor and a chief instructor who signs off on anything that touches reporting. Their names do not matter; their roles do.

Monday: the report that was never filed

A student pilot tries to submit a hazard report on a Monday morning. The form loads, the student fills it in, the confirmation page appears, and the safety manager never receives the email. A second report, filed the same afternoon, arrives three hours late. The safety manager does what every careful person in aviation does: write down exactly what is known. Two reports, two different delays, no error messages on the screen.

The contractor, who looks after the server alongside two other clients, receives that note. Their first instinct is to restart the mail service. Restarting before looking is filed by the Atlas under habits to avoid, since it wipes out whatever would explain the fault; the contractor holds back and instead pulls the journal, the queue status and the clock reading.

Tuesday: the clock tells a story

The journal shows something odd: entries from the web application and entries from the mail service disagree about the time by almost a minute. Reports are stamped with the application’s clock, delivery is attempted on the mail service’s schedule, and the retry logic keys off timestamps. A clock error does not produce a visible failure; it produces behaviour that looks random.

The Atlas treats time as a first-class operations concern. Its recommended setup uses the chrony service, with a setting that makes the clock slew around a leap second rather than jump, and a setting that allows a one-time step if the offset is large at start-up, limited to the first few updates. Its cautionary tale is the leap second of 1 January 2017, when software that assumed time never runs backwards computed a negative duration and misbehaved, affecting roughly 0.2 per cent of DNS queries at one large network. The contractor checks the time service and finds it was never enabled after a recent rebuild. The virtual machine has been drifting on its own.

Wednesday: the second cause

Fixing the clock reduces the delay but does not eliminate it. A good diagnostician tests the first fix against the evidence, and the evidence says some messages are still stuck. The contractor looks at the scheduled job that flushes the mail queue. It is an old-style cron entry containing a date in its command. In that scheduler an unescaped percent sign becomes a newline, so the job has been truncated since the entry was last edited.

The Atlas steers toward systemd timers for exactly this reason, and for three more: a timer can run a missed job after downtime, it can stagger start times, and it can trigger an alert unit when the job fails. The contractor writes a timer, tests it once by hand and keeps the old cron line commented out with a dated note, rather than deleting it.

Thursday: the review, evidence first

The academy’s culture of safety reporting turns out to be useful here. The safety manager asks for a short review in the same format used for hazard reports: what happened, what contributed, what controls exist and what residual risk remains. The contractor fills it in.

  • What happened: notification emails from the reporting form were delayed or lost for four days.
  • Contributing factors: an unsynchronised clock after a rebuild; a scheduled job truncated by an unescaped character; no alert on job failure.
  • Existing controls: none that would have detected either factor.
  • Residual risk: delayed visibility of hazard reports, the one outcome the whole system exists to prevent.

The chief instructor asks a pointed question: how would we have known? The honest answer is that nobody would have known until a student complained. That prompts the real work of the week.

Friday: turning findings into measures

The contractor proposes measures taken directly from the Atlas’s list of indicators: the success rate of scheduled jobs, monitoring coverage, and the number of machines that match a written baseline. For this one server the targets are modest. Every recurring job must report success or failure to a mailbox the contractor reads. The time service must be monitored, with an alert when the offset exceeds a defined threshold. A short baseline document lists the settings every rebuild must apply.

The contractor adds a fourth item that the safety manager finds especially valuable: a quarterly restore test. On backups the Atlas draws a line between a storage snapshot and a true backup, and the only proof of the second is a successful restore. Once a quarter, a sample of reports is restored to a scratch location and compared with the live data. The test is logged with its date, duration and result.

The numbers behind the decision

The contractor also does a quick piece of arithmetic that the safety manager appreciates, because aviation people like to see a margin written down. The disk holding the report attachments is 200 GiB, 150 GiB are used, and the last four weeks added 20 GiB. That is about 5 GiB a week, so the runway before the volume fills is comfortably measured in months, not days. Writing the sum on the baseline sheet means a future change in the growth rate is noticed at once rather than at the moment a photograph upload fails. It is a small, boring figure, and that is the point.

What the story shows

Three observations stand out, and each matches a principle in the reference.

  1. Quiet failure is the normal failure. Nothing crashed. Services were up and pages loaded. The failures were in time and scheduling, the two areas that rarely appear on an uptime chart.
  2. The first explanation is rarely the whole one. The clock was a real cause, but it was not the only cause. Testing the fix against the evidence found the second.
  3. Records beat memory. The five-rung maturity ladder in the reference starts with work that depends on memory and finishes with hosts that can be replaced on demand. The academy started the week at ad hoc: the server worked because one contractor remembered how. It ended the week with a baseline document, a timer, an alert and a scheduled restore test, which is clearly further up the ladder.

How a team like this can take it further

The reference sketches a three-month arc. Month one is inventory and the removal of any shared administrator login. Month two applies a baseline through configuration management, move legacy jobs to timers, cap log sizes and agree a restart policy. Month three adds a restore sample, a rehearsal of console recovery, a calendar of operating-system support dates and training for a second person on the triage routine. For a single-server academy, even the first month is a big step.

The wider lesson for aviation people is a familiar one: a safety system is only as dependable as the systems that carry its reports. If you want the full reference behind the details in this story, the Atlas page for SysOps lays out the scope, the practices and the cases in one place, and it is a sensible thing to hand to whoever looks after your own servers.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top