When a lesson is cancelled for weather or a simulator goes down for maintenance, students expect to hear about it quickly, usually by text message. That notification service is small, easy to overlook and surprisingly important: a missed alert means a wasted trip to the airfield. Site reliability engineering, abbreviated SRE, offers a method for keeping such a service dependable without a large team. This how-to walks through seven steps, in order, with the arithmetic shown so you can reproduce it.
What follows leans on a reference entry on SRE from the xOps School Atlas, adapted to a messaging service at a flight academy. It deals with administrative notifications only; operational and safety decisions are handled by their own procedures.
What you need before you start
- A log or report of messages sent, and of messages the provider confirms as delivered.
- A spreadsheet with one row per day.
- About an hour for the first pass and fifteen minutes a week afterwards.
- One person with the authority to say “we pause changes now.”
Step 1: Describe the journey in one sentence
Write down what a student should experience: “When the schedule changes, I receive a text within five minutes.” Keep it short and written from the student’s point of view. Leave the machinery out of it. A promise that needs a paragraph to explain is not yet ready for measuring.
Step 2: Decide what counts as a good message
Two conditions make a message good: it arrived, and it arrived on time. Count every alert that was meant to go out as a valid event. Count it as good only when the provider reports delivery within your five-minute limit. Treat slowness as a share of messages rather than an average, because a handful of alerts arriving after an hour will not show in a flattering mean. The Atlas also says that quick failures should count against you in an availability sense, since a student who gets an immediate error still did not hear the news.
Step 3: Take the readings from the student’s side
Delivery receipts help, but they come from the provider’s side. Add a test phone, or a colleague’s number, that receives a dummy alert every morning at a fixed time. Log whether it arrives and how long it takes. This external probe catches the failures your own system cannot see, such as a carrier delay that your logs record as a success.
Step 4: Set a target and calculate the allowance
Choose a target you can honestly meet. For a notification service, 99.5 percent delivered within five minutes over a rolling four weeks is a sensible starting point. Do the arithmetic: if the academy sends around 6,000 alerts in four weeks, the allowance is one-half of one percent, or 30 messages. Write both figures in your sheet. Four weeks is useful because it contains every weekday the same number of times, which stops busy Mondays from skewing the picture.
Do not be tempted by a target of 100 percent. The Atlas stresses that perfection is the wrong aim for nearly any service, since the bill for every further notch of dependability keeps climbing faster than the benefit. Phone networks alone make it unattainable.
Step 5: Write the rules for a low allowance
This step turns numbers into behaviour. Draft a short policy with two or three triggers, for example:
- If one incident uses more than a fifth of the allowance, hold a short written review with a named follow-up task.
- If the full four-week allowance is spent, pause every change except urgent repairs until it recovers.
- If the allowance is barely touched for several months, consider tightening the target or shipping improvements faster.
Get the policy initialled by whoever runs the academy. A rule without a signature tends to disappear the first time a deadline looms.
Step 6: Set up two kinds of alert
Watch how fast the allowance is being used. A simple version: compare the share of failures in the last hour with the share your target allows. If the last hour ran at fourteen times the permitted pace, someone should be called. If the pace over the past day is only about three times the permitted rate, open a ticket for the morning instead. These ratios come from the worked example in the Atlas for a 99.9 percent target; for 99.5 percent you can start with the same shape and adjust after a month of experience.
Avoid sending a message for every blip. Alerts people ignore are worse than no alerts at all.
Step 7: Measure the toil and try changes on a small slice
Time the chores that keep the service going: updating phone lists by hand, resending failed messages, copying templates. As a worked example, if two staff members each spend 80 hours across a quarter on the service and 60 of those combined 160 hours go on manual chores, the share is 37.5 percent. The Atlas suggests staying below half; once you cross it, automate the worst chore first.
When you change something, release it to a small proportion of recipients first. The source offers a handy calculation: say a faulty build reaches only one recipient in twenty and breaks a fifth of its messages; the overall error rate rises to just 1 percent, a far gentler hit than failing for everybody. Start with a few staff numbers, wait a day, then widen.
A sample weekly routine
Once the setup exists, upkeep is light. On Monday morning, open the sheet and add the previous week’s totals of valid and good messages. Update the running figure for the four-week window and note how much of the allowance is left. On Wednesday, glance at the test-phone log for slow or missing arrivals. On Friday, spend ten minutes on the toil list: did any chore appear more than twice this week? If so, add it to the automation shortlist.
Keep a one-line note beside each week, such as “carrier delay on Tuesday, provider ticket opened.” Those notes become the raw material for any review later. After three months they also let you answer questions like “does the service struggle on the first day of a new term?” with evidence instead of a hunch.
Common mistakes to avoid
- Measuring only the provider’s confirmation: add the test phone.
- Averaging percentages across days: sum the good messages and the valid messages first, then divide.
- Forgetting dependencies: your provider should be reliably better than your own target, because you inherit its failures.
- Never reviewing: put a fifteen-minute check in the calendar each week.
Where to go from here
After a month you will have a record of how the service really behaves and a rule for what to do when it slips. Revisit the target every quarter. If the allowance is always nearly full, the target may be too generous; if it is always empty, perhaps your standards are higher than students need.
The whole system fits on one spreadsheet tab and a single page of policy. It will not make the phone networks perfect, but it will tell you quickly when your own part is failing, and it gives the academy a calm way to decide what to fix first.
