Safety management teaches a particular way of thinking about trouble. You do not ask who is to blame; you ask what the system allowed. A report records the event, the contributing factors, the barriers that failed and the corrective actions. That structure suits technical faults just as well as flight events, so here is one imagined problem from container operations written up in that style. Practice points below are taken from the Ops Atlas (xOps School); the incident is invented for teaching and describes no real organisation.
The event
A small team keeps a reporting tool for an operator’s safety office on a single server. The tool runs in one container. Over three weeks it has vanished from the status screen at irregular times, mostly during a busy end-of-month period. Each time, someone restarts it and things recover. The team calls it a gremlin, which in safety terms is the word that says the investigation has not started yet.
Before going on, one clarification about vocabulary. The Atlas notes that ContainerOps is not a standard name and that others use it for different things. It uses the phrase for the care of individual containers below any orchestrator. To decide which questions were in bounds, the investigators consulted the Atlas page that defines ContainerOps.
Symptom one: an exit code that says exactly what happened
The first useful clue was the exit code. The container’s status showed 137. The Atlas decodes this as 128 plus 9, meaning the process received SIGKILL. That does not say who sent the signal. Two sources are possible: a deliberate stop that exceeded its grace period, or the kernel’s out-of-memory handling. The Atlas also cautions that on cgroup v2 an out-of-memory kill can reflect pressure on the whole host and not only on the container, so the investigators checked both.
Contributing factor: no memory limit
The container had no memory ceiling. During month-end, the tool loaded larger data files, and its memory use climbed until the host ran short. The Atlas is firm that memory and process limits should always be set. With a limit, the kernel would have acted on that single container at a predictable point, and the rest of the host would have been spared. Without it, the container shared in a general scramble for memory and was the one that got cut.
Contributing factor: a clumsy stop
The second problem appeared when the team tested a normal restart. It took ten seconds and ended with another 137. The entrypoint was written in shell form, so the program ran underneath a shell wrapper, and the polite termination signal never reached it. After the grace period the container manager, which in Compose defaults to ten seconds, sent the hard kill. The Atlas lists this pattern and the cure: use the exec form, have the program handle the signal and add the init flag.
The consequence mattered. A hard kill during a write can leave a half-finished record. In a safety reporting tool that is a data-quality hazard worth taking seriously.
Symptom two: slow responses before each failure
Users also reported sluggishness in the hours before a disappearance. The investigators suspected processor throttling because someone had added a CPU limit months earlier. The Atlas explains the check: compare throttled periods with total periods. Its worked case shows a container limited to two CPUs that averaged only 1.5, and still found 240 of 600 periods throttled, a rate of forty percent. Where response time matters, the Atlas would like to see that ratio stay beneath five percent.
Here the numbers were milder, yet a bursty report generator was clearly hitting the cap. The team loosened the limit and kept the memory ceiling in place.
Symptom three: a disk that quietly filled
The third finding was unrelated to the failures but would have caused the next one. The server’s disk was nearly full. The container logs were the largest single item, since json-file logging, in its default form, never rotates anything, a gap the Atlas calls out. A full disk would have taken every service down at once. This is the sort of latent condition safety management exists to find before it matters.
What the first response got wrong
A fair report also reviews the response, and here it is instructive. Each time the tool vanished, the person on duty restarted it and moved on. That was sensible for the moment and harmful for the record, because the restart erased the evidence. No one saved the container’s final state, the exit code or the kernel messages for that hour. Safety offices call this a lost learning opportunity.
The second shortfall was a leap to a favourite theory. Someone had read that “containers crash because of bad code” and spent two days searching the application for a defect that did not exist. The data pointed elsewhere from the start: the exit code, the missing memory limit and the shell-form entrypoint were all visible in minutes. The Atlas lists a related habit among its anti-patterns, patching running containers through exec, and the team had tried exactly that on one occasion, which accomplished nothing because the next restart discarded the change.
The third was the absence of a simple, shared checklist. A short list taped beside the status screen would have prompted the right sequence: capture the exit code, check memory, check the stop path, check throttling, check disk.
Barrier analysis
The report then asked which barriers should have caught the problem.
- Resource limits: absent for memory, too tight for CPU bursts.
- Clean-stop design: absent, because the shell wrapper hid the signal.
- Monitoring: the team tracked uptime, not exit codes, so the pattern stayed invisible.
- Log housekeeping: not configured.
- Ownership: the tool had no named owner, which the Atlas lists among the anti-patterns of estates that quietly age.
Corrective actions
The actions were deliberately small, specific and dated:
- Set a memory ceiling informed by month-end peaks and a process limit alongside it.
- Switch the entrypoint to exec form, handle the stop signal, add the init option and verify a stop completes quickly.
- Revisit the CPU cap and watch the throttle ratio weekly for a month.
- Rotate logs by size and count.
- Name an owner and record the image’s digest in the tool’s inventory entry.
- Track OOM kills against container-days of uptime, one of the Atlas metrics, and look at the trend each month.
Why the fix has to be repeatable
One more point is worth making. The team’s first instinct was to repair the running container and carry on, but the Atlas approach is to rebuild the image with the corrected settings and redeploy it by digest. That sounds heavier, yet it produces something a safety office values: a change that can be described, reviewed and repeated. If the problem returns, the team can compare two exact builds rather than argue about what somebody typed into a terminal last month.
It also supports a clear rollback. Because the earlier digest still exists, going back is a matter of naming it. No one has to remember what the previous version was called or hope that a floating tag still points where it used to.
Lessons for the safety office
Three observations apply well beyond this one tool. Failure signals such as exit codes are free, but only if somebody reads them. Defaults are not decisions; the missing log rotation and the absent memory ceiling were the result of nobody deciding anything. And the most useful repair was an organisational one, a named owner, since an estate without owners ages in silence.
There was also a short follow-up check at ninety days, as the Atlas plan suggests for such programmes: confirm that the changes held, run a restart drill, and record the results. A repeat occurrence would have meant the analysis missed something, and the process would have started again.
Safety culture rewards the team that reports small things early. A container that vanishes twice a month is a small thing until the month it takes a day of records with it. The tools to understand it were already on the server; what was missing was the habit of asking the system what it allowed.
