GitOps Drift: A Method for Finding the Cause

Aviation maintenance never treats a warning light as the fault; the light is the symptom, and the job is to trace back to the component that set it off. Drift deserves the same respect. Teams that run their platforms under GitOps — declared configuration in a versioned store, an agent continuously pulling and reconciling the live environment — eventually meet the problem this article is about: the live environment disagrees with the declared one, repeatedly, and nobody can say why. The instinct is to fight the symptom, resyncing by hand until the alerts stop. The discipline is to diagnose. What follows is a working diagnostic method, evidence-led, arranged as problem first, cause second, fix third.

A word on stakes before the method. Drift that recurs is drift that has already been paid for at least twice — once when the unrecorded change landed, and once when the reconciler undid it. Every additional round trip costs another investigation, another late evening, and another notch of confidence lost in the platform. The economics argue for an hour of diagnosis over a month of clearing the same alert.

The Problem: What GitOps Drift Actually Looks Like

The presentation is familiar. Alerts fire because a field in the live cluster no longer matches the repository. Someone clicks resync. The alert clears. Days later the same resource diverges again, sometimes in the same way, sometimes wearing a new shape. The team starts describing the platform as “flaky”, which is the moment the diagnosis is usually lost — because a reconciler that keeps finding the same difference is not malfunctioning. It is reporting one, in the only language it has.

The healthiest reframe: every repeated drift event is a message from your own infrastructure saying that something in the environment can change which you have not declared. Treat each recurrence as data, not as noise.

The Diagnostic Sequence, Run in Order

  1. Freeze the reflex. Ban manual resyncs for one day. Every forced resync destroys evidence by overwriting the difference you needed to study.
  2. Capture the difference. For each drifted resource, record what the repository says, what the cluster says, and when the divergence first appeared. Timestamps matter more than contents.
  3. Correlate with human activity. Cross-reference divergence times with deploys, on-call changes, and who was logged in. Most drift clusters around human convenience.
  4. Classify each difference. Approved-but-unrecorded, unapproved-but-necessary, unexpected-but-benign, or hostile. The four classes have four different remedies.
  5. Attribute the owner. Which actor — script, colleague, controller, vendor agent — wrote the change. Reconciler event logs usually answer this faster than git history can.
  6. Only then fix. By now the repair is specific rather than general, which is the whole point of diagnosing before touching anything.

The classification step deserves a note, because teams skip it and lose the plot. An approved-but-unrecorded change indicts the workflow; an unapproved-but-necessary one indicts the on-call tooling; a benign surprise indicts a controller configuration; a hostile one indicts your access list. Four classes, four different owners, and conflating them is how organisations end up “fixing” a permissions problem by tuning alert thresholds.

Evidence From One Measured Week

Why go to this trouble? Because the unmeasured version of this problem is consistently worse than teams assume. Consider a week from a lab exercise that measured a small estate running the pull-based model end to end. One application in five failed to apply cleanly — a 20% apply failure rate, far above what the team guessed before measuring. The estate spent 1.12% of all application-minutes out of sync for the week, and just two drift events accounted for 94% of that time; the long tail of tiny divergences was noise by comparison. Of six drift events, four were reverted through the normal reviewed path, and — the finding that embarrassed everyone — four of the five out-of-band changes recorded that week had no ticket anywhere.

Read those numbers the way a safety board would. The headline problem is not the automation misbehaving; it is unrecorded human activity moving through the environment faster than the process can see. Measurement relocated the blame from the tool to the workflow, which is precisely what a good diagnosis is for — and note how little of the week’s trouble any amount of extra resyncing would have solved.

Root Causes Behind Most GitOps Drift

Run the sequence above across enough incidents and the same small set of culprits keeps appearing. None of them are exotic; all of them are embarrassingly common in production estates that consider themselves well run:

  • Mutable references. A deployment pinned to latest, a chart version left floating, a remote base tracked to a moving branch head — each one invites the live system to become something the repository never described. Immutability is not pedantry; it is the precondition for the whole model.
  • Controllers with their own opinions. Autoscalers rewriting replica counts, webhook-driven tools filling in fields at runtime. These differences recur forever unless the reconciler is told, field by field, to leave them alone.
  • Cascading deletions. An application generator that owns dozens of child applications turns one mistaken removal into a fleet-wide pruning event. Generators need deletion guards the way fuel trucks need earthing cables.
  • Self-healing at war with incident response. The reconciliation loop faithfully undoing an emergency mitigation because nobody paused it. Any automatic corrector needs an equally well-rehearsed pause mechanism.
  • Unowned access paths. Someone — or some assistant, script, or integration — with write access to a watched path is, functionally, holding deploy credentials, whether the team listed them or not. Inventory those paths before you assume you know who can change production.

The Fix, Applied in the Right Order

Before the ordering, one warning drawn from the maintenance bay: resist the urge to fix while diagnosing. Every premature change invalidates the observations you have not finished taking, and the investigation restarts from zero. Diagnose completely, then repair deliberately, in the order below.

Order matters, because the tempting fix — tighten the alerts — treats the symptom once again. First, make the legitimate-but-unrecorded changes recordable: if people keep making the same out-of-band change because the approved path is awkward, the approved path is the defect, and it needs widening until the awkward path stops being attractive. Second, decide drift policy per resource: reversion for unapproved changes, documented tolerance for controller-written fields, alerting for the ambiguous middle, and protection from pruning for anything whose loss would be unrecoverable. Third, pin everything and remove mutable references on a schedule, not one crisis at a time. Fourth, add deletion guards around anything generated at scale. Fifth, rehearse the pause: the person handling the next incident must be able to stop the reconciler without asking permission in a burning room.

Verifying That the Diagnosis Held

Close the loop the way the lab exercise did: measure the same quantities for a comparable week afterwards. Apply failures should fall first — they respond quickly to pinned references. Out-of-sync time should concentrate into a small number of understood events rather than a scatter of mysteries. The unticketed-change count should approach zero; if it does not, there is an access path or a workflow you have not found yet. And reconcile the human side too: the team should be able to name, for every divergence in the new week, which class it belonged to and who owned it. That is the practical definition of drift under control — not zero divergence, which no honest environment achieves, but zero unexplained divergence.

One last calibration for sceptical readers: none of this evidence comes from a vendor’s slide deck. It comes from measuring a real (if small) estate and publishing the unflattering numbers alongside the good ones. Hold your own programme to that standard — count, attribute, classify — and the drift warning light becomes what it always should have been: a prompt to find the component, not a reason to hit the panel with a screwdriver.

For the definitions, the reconciliation model and the wider practice this diagnostic assumes, the GitOps guide hosted by xOps School is the reference this method was written against.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top