My Team’s Airflow to Prefect Migration: 200 DAGs, 18 Months, and a Lot of Scar Tissue

We migrated 200 Airflow DAGs to Prefect in a HIPAA environment. It took 18 months, not the 9 we planned. Here’s what actually went wrong.

I never thought I’d be writing an airflow to a prefect migration guide that enterprise teams could actually trust. Mostly because every time I searched for one during our own migration, the results were either marketing fluff or suspiciously smooth success stories. Our reality? It looked nothing like those. Years had gone into building on Airflow, and then we doubled our original migration timeline once we decided to move to Prefect in a HIPAA-regulated environment. Nothing about it was clean. But we ended up with a system we actually like, plus a lot of scar tissue others can learn from.

This article walks you through how our team moved over 200 Airflow DAGs to Prefect inside a healthcare analytics environment. I’m going to lean heavily on what worked, what imploded, and the healthcare-specific challenges where prefect vs airflow healthcare data pipelines behave differently. You’ll see patterns, anti-patterns, and the parts of the process that caught even our most experienced engineers off guard. Think of this as a perfect migration case study data engineering folks can hand to their leadership when someone asks, “Why is this taking so long?”

The Audit Phase: Categorizing DAGs into Migrate, Refactor, or Kill

Our first shock came fast. Over 200 DAGs existed in our Airflow environment, but only about half were actually doing anything useful. Honestly, I shouldn’t have been surprised. This pattern shows up across organizations constantly. If something’s easy to schedule, people schedule it forever.

Three buckets emerged from our categorization:

• Migrate exactly as-is
• Refactor or merge
• Kill without guilt

That kill list? Cathartic. Dead tables, deprecated models, pipelines with owners who’d left the company ages ago. Healthcare data teams tend to accrete workflows around compliance requests, so churn is normal. What surprised us was how many DAGs existed only to copy files between buckets or convert formats. The prefect handled those much more cleanly, so having this inventory early helped us cut scope dramatically.

For anyone wondering how to migrate from Airflow to Prefect in healthcare settings, start with an audit. It saves months. It also prevents you from dragging legacy failures into your shiny new orchestrator.

HIPAA in the Cloud: Architecting for Perfect Healthcare Data Compliance

HIPAA was the part everyone feared. And honestly? Fair enough. Airflow had been self-hosted, sitting inside our VPC, hardened over the years. The prefect introduced new moving parts, and compliance officers get twitchy when anything moves.

Prefect Cloud got evaluated, but ultimately we went with a self-hosted Prefect Server deployed inside our cloud environment. That choice removed entire categories of approvals we never wanted to request. (If you’ve ever had to explain container orchestration to a compliance committee, you know exactly what I mean.)

Here’s what our architecture looked like:

• Self-hosted Prefect Server inside a private subnet
• Cloud-based artifact storage with server-side encryption
• Dedicated encryption keys for orchestration metadata
• Audit logging piped into our existing security monitoring pipeline
• Workers running on container services with strict access controls

Healthcare data pipeline orchestration best practices don’t differ much from other regulated industries, but the nuance is in how metadata is handled. Prefect labels and parameters sometimes contain PHI if engineers aren’t careful. So we added linting rules and a simple validation script to block unsafe variable names.

Small simulation tests became my usual habit. Random payloads mimicking PHI shapes got pushed through just to see what leaked to logs. The prefect behaved nicely once we fixed the default logger levels.

The Strangler Fig Approach: Running Dual Orchestrators Without Losing Your Mind

Running Airflow and Prefect at the same time was always part of the plan. Killing Airflow instantly would’ve been reckless.

A strangler pattern got stitched together like this:

• All new pipelines built in Prefect
• Existing Airflow DAGs frozen for new features
• Migration candidates mirrored in the Prefect for several weeks
• Prefect flows wrote sanity check signatures to a lightweight data store
• Airflow DAGs wrote their own signatures
• A small Python service compared the outputs

Sounds complex, right? It was. But it prevented risky cutovers. When teams ask why switch from Airflow to Prefect 2026, my honest answer is that you rarely switch in a single event. You transition, compare, then slowly shut down the old system when nothing screams.

Human factors proved hardest. Engineers forgot which orchestrator owned which job. Sound familiar? Documentation helped, but labeling dashboards helped more.

DAG to Flow Translation Patterns: What Converts Cleanly and What Requires Redesign

Here’s the thing. Airflow and Prefect feel similar only at first glance. Once you migrate, the differences become loud.

What migrated cleanly:

• Pure Python tasks with no XCom usage
• Short linear DAGs with stable dependencies
• File movement or metadata ingestion pipelines
• Simple daily schedules

What needed redesign:

• Heavy XCom DAGs using pandas DataFrames
• Dynamic task mapping done through for loops in PythonOperators
• Complex DAGs with cross-DAG triggers
• Anything using Airflow’s SLA feature

In a healthcare context, prefect vs airflow healthcare data pipelines differ most in how they handle dynamic branching. Prefect’s mapping felt more natural, but a few pipelines still needed rework because they relied on Airflow’s implicit state handling.

Engineers kept asking for step-by-step guidance on migrating airflow DAGs to prefect flows. So here’s my rule of thumb:

1. Rewrite operators as functions
2. Convert DAG parameters to Prefect variables
3. Replace Airflow-specific utilities with native Python
4. Add logging and retry behavior explicitly
5. Run a small batch through staging
6. Validate outputs against the Airflow version

A few pipelines actually got simpler. A few became messier before they became simple. That part’s normal.

The Hidden Costs: Retraining, Observability Gaps, and Vendor Lock-in Tradeoffs

Planning covered two categories of work: engineering time and infrastructure. What surprised us? Retraining costs nobody anticipated. Prefect is friendlier for Pythonistas, but Airflow veterans who relied on UI-centric debugging had a slower ramp.

Observability felt different, too. Airflow’s graph view is clunky, but teams knew how to read it. The prefect’s timeline view took time before people trusted it. For several weeks, both UIs ran side by side just so analysts felt safe.

Vendor lock-in questions came up often. Even though Prefect was self-hosted, parts of the runtime felt more tied to the framework than Airflow ever did. I don’t think this is bad. But it’s a tradeoff teams should make intentionally.

Common pitfalls during airflow to prefect migration projects also hit us, like pipeline owners assuming backward compatibility where none existed. Every translation needs testing. Every. Single. One.

Post-Migration Reality: What We’d Do Differently

Looking back several months later, a few things stand out.

Starting the audit earlier would’ve helped. Killing even more DAGs would’ve helped. Adding a translation library for common Airflow patterns would’ve helped. And setting expectations that this isn’t a lift-and-shift, it’s a redesign, would’ve saved headaches.

Biggest win? The prefect made local development faster. Engineers finally stopped SSH-ing into Airflow workers to debug things. Small joy, big productivity gain.

Biggest miss? The documentation load was underestimated badly. People needed more guidance, especially junior engineers new to workflow orchestration tools and healthcare analytics comparison topics.

And if you’re wondering, yes, we still occasionally find an old Airflow DAG running quietly in production with no owner. Healthy reminder that migrations are never totally done.

If you’re preparing your own airflow to prefect migration guide that enterprise teams can trust, here’s the checklist I wish we’d had on day one.

Migration readiness checklist:

• Do you have a complete DAG inventory?
• Are you killing at least a third of them?
• Do you know which ones contain PHI risks?
• Is Prefect deployed in a compliant architecture?
• Is your team trained on Prefect before migration starts?
• Do you have a dual-run strategy?
• Are outputs validated during the transition?

Go or no-go criteria:

• If your DAGs rely heavily on cross-DAG triggers, delay migration.
• If your team lacks Python fluency, train them first.
• If compliance reviews aren’t ready, stall.
• If Airflow is heavily customized, prepare for refactoring.

Migrations expose every awkward corner of your data platform. And that’s fine. Because the lessons learned migrating workflow orchestration tools usually make your system cleaner and your team sharper.

If you’re about to start, take a breath. Plan longer than you think. And don’t delete your Airflow cluster until you’re absolutely sure you don’t need it.