After 47 Post-Mortems, I Can Predict the Problem Before the Meeting Starts

47 post-mortems, 7 patterns. I can predict why your data pipeline failed before the meeting starts. #3 is the one nobody admits to causing.

I’ve lost track of how many post-mortems I’ve sat through where the room slowly realizes the same thing: the real failure wasn’t the outage or the broken metric. It was the set of quiet gaps everyone thought they could deal with later. These gaps are so common that I can almost predict them before the meeting starts. They map surprisingly well to the data engineering best practices that enterprise teams overlook, even when they think they’re being mature.

Ask engineers why enterprise data projects fail, and you’ll get poetic answers about culture or communication. But when you actually run autopsies? You start to see a more mechanical pattern. Seven missing practices show up again and again, almost like a signature left on every broken pipeline, stalled migration, or unusable warehouse.

I’ve seen these gaps at companies with thousands of engineers and at startups pretending to be enterprises. I’ll admit it: I caused some of them early in my own career. My background in adaptive experiments taught me a lot about theory, but real production failures taught me how data systems behave under stress.

Let’s walk through the seven patterns I see most often, why they hurt, and how you can avoid becoming the next example file in someone’s onboarding deck.

1. Schema Evolution as First-Class Infrastructure

We all tell ourselves this lie: schema changes happen slowly. We can coordinate them manually. Teams will be careful. Right?

Production tells a different story. A product team adds a field at midnight. Someone else renames an enum value during a hackathon. A third team backfills data in a way that reorders timestamps. By Monday, half your dashboards are flatlined, and nobody knows why.

Enterprises tell me they follow enterprise data architecture best practices, 2025 style. Yet schema evolution is usually a GitHub issue plus a prayer.

Strong systems treat schema evolution like a core service:

• Automated compatibility checks for every commit
• Versioned schemas with programmatic enforcement
• Shadow read and write paths to catch surprises
• A publish step that blocks if downstream consumers break

At large tech companies, teams often test schema changes with synthetic data before rollout using simulation tools, mostly because engineers get tired of waking up to dashboards on fire. These tools can save weeks of debugging over time.

When your company thinks schemas can be managed socially, you’ve already started the countdown clock to your next incident.

2. Data Contracts with Teeth: Enforcement Mechanisms That Actually Work

Data contracts sound great in kickoff meetings. Everyone nods. Everyone agrees. Then? Nothing happens.

A contract without enforcement is just a PDF nobody reads. Enforcement without observability is just a blacklist you’ll regret two months later.

Teams often ask me how large companies structure data engineering teams to make contracts work. Short answer: They give teams tools that act like circuit breakers:

• Producers can’t publish non-compliant data
• Consumers can’t silently continue reading malformed data
• Owners are automatically notified when something drifts
• Failing contracts trigger pipeline slowdowns or holds, not Slack arguments

When I ask enterprises for an example of a real contract violation caught early, most can’t give one. That tells me everything. Your data contracts never fire? They probably aren’t connected to anything.

3. Pipeline Idempotency Beyond the Happy Path

I can spot a fragile pipeline from a mile away. It’s the one with a comment like this:

# Safe to assume this only runs once

Sure. Until it doesn’t.

Some of the most painful data engineering mistakes enterprise companies make come from half-written idempotency. Teams handle retries for the easy cases, but forget:

• What happens after a partial write
• What happens if a downstream system commits while the upstream fails
• What happens when retries collide with cleanup jobs
• What happens when a backfill runs against historical data that behaves differently

Idempotency isn’t just rerunning a job. It’s being able to reconstruct the truth regardless of execution order. Your DAG can be put into a weird state by a single operator retry? You’re one on-call shift away from your next rewrite.

I’ve lived the pain of reconstructing metrics with dozens of conditional JOINs because a pipeline once wrote duplicates. Never again.

4. Observability That Answers Why, Not Just What

Most enterprises now have dashboards showing pipeline success rates. Great. But honestly? That answers almost nothing.

When something breaks, the real questions are:

• What changed?
• Where did it change?
• Which data moved?
• Which consumers were affected?
• Did semantics drift or did structure drift?

Answering these requires lineage plus semantic monitoring. Lineage gives you the path. Semantics tell you the behavior changed.

My preference is for systems that:

• Track field-level lineage
• Monitor statistical properties of key columns
• Alert when distributions shift unexpectedly
• Provide rewindable playback for debugging

In experimentation work, semantic monitoring of treatment assignment and exposure can save teams from pushing bad models to millions of users. Without it, you’d be guessing. Guessing isn’t a strategy.

Your observability only tells you success or failure? You’ll spend your nights reading logs and whispering to the data warehouse like it’s a stubborn mule.

5. Governance at the Pipeline Layer, Not Just the Warehouse Layer

Most enterprises focus governance on the warehouse. It’s the place auditors understand, the place leadership cares about, the place where compliance lives. But here’s the problem: by the time data reaches the warehouse, the real errors have already happened.

Companies keep investing heavily in data engineering governance frameworks for enterprise work, but almost none of them govern pipeline code, pipeline configs, or pipeline transformations. That’s where lineage breaks, PII sneaks in, and logic gets rewritten without review.

Governance needs to exist where change happens:

• Field-level tagging is enforced at ingest
• Access controls tied to pipeline ownership
• Automated checks that block pipelines from outputting restricted fields
• Mandatory review rules for transformation logic

Governance kicks in only after data is stored? You’re just catching the wreckage, not preventing it.

6. Capacity Planning for Data Gravity

Every enterprise eventually reaches a moment where the data becomes immovable. Migration plans fall apart. Streaming jobs back up. Query costs spike. Engineers start talking about the warehouse in the same tone people use for ancient monuments.

Data gravity happens when:

• Data volume explodes faster than storage planning anticipates
• Cross-region traffic becomes unaffordable
• Legacy systems never get decomposed
• New pipelines keep attaching to old tables because they’re convenient

I’ve seen companies try to move petabytes with scripts written in a hurry during an outage. That’s how you end up with corrupted tables, angry executives, and a multi-quarter rewrite.

Capacity planning isn’t a finance exercise. It’s engineering. Ask why enterprise data projects fail during scaling moments, and this is usually the reason: the system assumed infinite bandwidth and infinite patience.

Teams should build:

• Forecasts for storage, compute, and I/O
• Automatic throttling mechanisms
• Regional replication plans are designed up front
• Migration windows with rollback paths

Data becomes heavy. Treat it like it will.

7. Cross-Team Dependency Mapping Before It Becomes Archaeology

At a certain point, your data ecosystem becomes a city built on top of older cities. Pipelines depend on pipelines that depend on pipelines nobody owns.

One post-mortem sticks with me. Root cause? A table was deleted because the owner thought it was unused. Turns out it fed a number of undocumented jobs that nobody had touched in years. Those jobs fed an executive dashboard. That dashboard fed a weekly meeting that decided product funding. Nobody knew the chain existed. You can imagine how that conversation went.

Older companies face this more often, and they’re more likely to ask why enterprise data projects fail and how to fix them. Dependency blindness is usually the answer.

You need:

• Automated mapping of pipeline dependencies
• Identification of orphaned datasets
• Ownership metadata tied to escalation paths
• Regular cleanup cycles

Otherwise, your organization becomes a data archaeology site, complete with abandoned tables and mysterious metrics carved into stone.

Run a quick fifteen-minute audit to see whether your team faces these risks. Ask yourself:

• Do schema changes have blocking checks, or just guidelines?
• Are data contracts ever enforced automatically?
• Can every pipeline be safely rerun from scratch?
• Can your observability answer why something changed, not just that it changed?
• Does governance apply before data hits the warehouse?
• Do you have forecasts for data volume and compute needs?
• Can you map dependencies without grepping the entire codebase?

Any answer that’s no means you’re ignoring the data engineering best practices that enterprise teams overlook until something breaks publicly.

Want a starting point? Pick one failing from your last incident report and trace it upstream. You’ll almost always find the missing practice hiding there.

Fix that. Then fix the next one. Some organizations figure this out. Most keep learning the hard way.