People who work with safety management systems have a healthy suspicion of reassurance. A spotless report can mean all is calm, or it can mean nobody went looking. That instinct is exactly what the data world needs, because a lot of what gets said about DataOps is comforting and wrong. Below are six myths, each tested against the mindset of someone who has sat through an occurrence review, with a plain correction after each.
Myth 1: “This is software delivery with extra tables”
This one is popular because it sounds tidy. The Ops Atlas tackles the claim head-on in its DataOps reference page. DevOps also runs and monitors production, so the difference is not that one touches databases. The difference is that data goes wrong on its own. A deployment pipeline protects the moment of change, but a data feed can break with no deployment at all: a supplier alters a column, a day’s file arrives half empty, a code list changes meaning.
The correction: treat the daily arrival of data as an event needing its own checks. In safety terms, you inspect not only after a modification but before every departure.
Myth 2: “The job ran cleanly, so the numbers must be right”
Every investigator knows that a process completing normally proves the process completed. It says little about the outcome. The Atlas borrows an idea from reliability engineering: each stage of a chain may announce success while records vanish in the gaps between stages. A job that copies zero rows will often finish with a success status.
The correction: check outcomes. Ask whether the newest record is recent, whether the amount looks sensible and whether the key fields are populated. Status lights describe machinery, not content.
Myth 3: DataOps is a product you can buy
The Atlas is plain about this: it is a practice and not a product. It also notes that an industry analyst still labels the tooling market as emerging and expects features to be absorbed into larger platforms, such that stand-alone tools risk looking redundant. Buying software does not give you ownership, agreements or an incident log. A reporting tool bought without a reporting culture is just an expensive filing cabinet.
The correction: start with habits that need no purchase: version control, a few plain checks, named owners and a written record of what went wrong.
Myth 4: More tests is always safer
A common reaction to an incident is to add checks everywhere. The Atlas lists a specific anti-pattern here: thousands of warn-only tests that bury the handful that matter. Warnings nobody reads are, in effect, publishing bad data with a disclaimer. Anyone who has watched crews tune out a nuisance alert that sounds every flight will recognise the pattern. Over time the sound becomes furniture, and the one real warning among them arrives as just another beep. Data teams drift into the same habit when dashboards are full of amber items nobody owns.
Noise has a mathematical side too. The Atlas calculates that a three-sigma alarm has a 0.0027 chance of a false positive on any single point, so a system running hundreds of such checks each day raises several false alarms every week. Pick checks that guard decisions, and mark some as blocking so that a failure stops publication.
The correction: fewer, sharper checks, with a clear rule on which ones can halt a release.
Myth 5: A schema check catches everything
Confirming that a column exists and holds the right type is useful but structural. The Atlas gives an example of a status value that keeps its name while its meaning changes. The format check passes while the reports mislead. A parallel from aviation would be a coding change in an occurrence category: the field validates, yet comparisons across years quietly lose meaning.
The correction: pair structure checks with written agreements between data producers and users covering meaning, quality rules, timing and a named owner.
Myth 6: A smart tool will learn what normal looks like
Automatic baselines are attractive, and they have a flaw: a slow decline gets learned as the new normal. The Atlas warns that learned baselines can absorb gradual deterioration, and that tests generated automatically describe data as it currently is, which means they need human review. A machine that studies a degrading situation will conclude that degraded is typical.
The Atlas also shows the seasonal version of the problem. A three-sigma band assumes one stable process, but weekly cycles are two. In the Atlas’s lab exercise, a band computed over all days let a short load slip by, yet a band computed from weekdays alone flagged it.
The correction: review baselines on a schedule, separate weekday and weekend behaviour, and have a person read the generated tests.
Why these myths survive
Each of these beliefs survives because it is partly right. Software delivery practices really do transfer, and success statuses really are useful, and tools really do help. The error is in treating a useful signal as a guarantee. In a safety setting, the habit of asking what a signal does not tell you is the heart of critical thinking, and it takes only a minute per dataset. Ask of every check what failure it would miss, and write the answer next to it. The gaps you list become the backlog for the next quarter, ordered by how badly they could hurt whoever relies on the number.
What a safety mindset adds
The common thread across the six myths is false comfort. Safety culture counters it with a few durable habits, and each transfers directly:
- Report and record. Keep an occurrence log for data: what happened, who noticed, and for how many days it ran unseen.
- Measure detection. Track how often problems are first spotted by users rather than by your own checks.
- Tier your attention. Critical data gets blocking checks and an on-call contact, while exploratory work gets little more than version control.
- Review on a rhythm. A short weekly look at the log beats a heroic annual cleanup.
A caution on scope
None of this replaces a formal safety management system, and it does not touch regulatory obligations. According to the Atlas, DataOps should not be mistaken for governance or for security wearing a different badge. What it can do is keep the numbers feeding your reports from betraying you. If your dashboards drive decisions about training or maintenance, the care you give them should match the care you give any other record.
Start small. Pick the one dataset you would least like to be wrong, write down who owns it, and add a recency check and a record-count check this week. Then watch for the first surprise, which rarely takes long to appear.
