Data migration: how to get it right (engineering guide for 2026)

Most data migrations underdeliver. Industry data consistently shows 50-70% of data migration projects exceed budget, timeline, or both — and a substantial fraction fail outright, requiring rollback to source systems. The technology isn't the issue; data migrations fail on scoping discipline, data quality assumptions, and stakeholder alignment.

This article maps what production data migrations actually require — the four migration types, the seven-stage process that ships, common failure modes with mitigations, and the toolchain we deploy. For broader treatment of data engineering economics, see /services/data-engineering and our data audit article.

What data migration actually means

Data migration is the process of moving data from one storage, format, or system to another. Sounds simple — execution is rarely so. Production data migrations involve:

• Extracting data from source systems (often legacy, often poorly documented)
• Transforming data to target schemas, formats, and quality standards
• Loading data into target systems with validation
• Cutover orchestration to minimize business disruption
• Reconciliation to confirm completeness and correctness
• Decommissioning source systems (or maintaining hybrid for transition periods)

Each stage has technical complexity and organizational coordination requirements. Most migration failures trace to inadequate planning of one or more stages.

Four migration types and their cost profiles

1. Storage-level migration

Moving data between storage systems with same logical structure — different filesystems, storage tiers, cloud regions, on-prem to cloud. Typically lowest complexity.

Examples: moving an existing relational database from on-prem to AWS RDS, S3 storage tier optimization, lift-and-shift cloud migration.

Cost: $30K-$200K depending on data volume and downtime tolerance.

2. Database engine migration

Moving from one database technology to another (Oracle to Postgres, MongoDB to DynamoDB, MySQL to BigQuery). Logical structures change but business data semantics remain.

Examples: Oracle to Postgres for cost reduction, MongoDB to PostgreSQL for relational requirements, NoSQL consolidation onto a single platform.

Cost: $100K-$500K depending on schema complexity and feature parity requirements.

3. Application platform migration

Moving data between application platforms with different data models. Most complex — business logic changes alongside technology stack.

Examples: Salesforce migration to Microsoft Dynamics, ERP consolidation (SAP to NetSuite), CRM platform transitions.

Cost: $250K-$2M+ depending on customization, integration count, and business process changes.

4. Modernization and transformation

Migrating to enable new capabilities — moving from legacy data warehouse to modern data lake/lakehouse, transitioning from batch to streaming, building cloud-native data platforms from on-prem foundations.

Examples: Teradata to Snowflake, Hadoop to Databricks, on-prem ETL to dbt + cloud warehouse.

Cost: $500K-$3M+ for enterprise platform modernizations.

Seven-stage process for production data migrations

Stage 1: Discovery and inventory

Before any migration planning, conduct a data audit to map the actual data landscape:

• Source system inventory with volumes, schemas, growth rates
• Data quality baseline measurements
• Compliance and governance posture
• Business process dependencies on each data asset
• Integration points with downstream systems

The discovery work routinely takes 4-8 weeks for enterprise migrations. Skipping it is the most common reason migrations overrun.

Stage 2: Target architecture design

Design the target data architecture with clear principles:

• Schema design — direct lift-and-shift vs. data model improvement vs. complete redesign
• Storage strategy — single platform vs. polyglot, hot/warm/cold tiering
• Access patterns — query workload analysis informs architecture decisions
• Compliance architecture — how regulatory requirements get implemented in target
• Performance targets — latency, throughput, scalability requirements

The architecture decisions made here determine 70%+ of migration success. Architecture compromises propagate through all subsequent stages.

Stage 3: Migration strategy selection

Three primary migration strategies, each with different trade-offs:

Big bang. Migrate everything in one cutover window. Lower complexity but higher risk — if it fails, you're rolling back the entire migration. Best for small, simple migrations.

Phased migration. Migrate domain by domain, system by system, over weeks/months. Lower risk per phase but longer total timeline. Most enterprise migrations use this approach.

Parallel run. Source and target systems both operational, with data flowing to both. Highest cost (running two systems) but lowest risk — instant rollback option. Best for high-stakes regulated migrations.

The right strategy depends on risk tolerance, downtime budget, and business continuity requirements.

Stage 4: Data preparation and cleaning

Data quality work that makes or breaks migrations:

• Deduplication of records that have multiple representations across source systems
• Normalization of formats, units, encoding
• Reconciliation of conflicting records across systems
• Standardization of values (codes, abbreviations, taxonomies)
• Enrichment of incomplete records where needed
• Filtering of obsolete or out-of-scope data

This stage routinely consumes 40-60% of migration project time. Underestimating it is the most common cause of timeline overruns.

Stage 5: ETL/ELT pipeline development

Engineering the actual migration pipeline:

• Extract from source systems — handle source system load, throttling, error recovery
• Transform to target schema — implement business rules, mappings, validation
• Load to target — handle target system constraints, transactional integrity, idempotency
• Validate during load — automated quality checks, anomaly detection
• Monitor progress — observability into long-running pipelines

Modern tooling (dbt, Airbyte, Fivetran, Databricks, Snowflake) accelerates this work substantially compared to custom-built pipelines.

Stage 6: Cutover and validation

The critical execution window:

• Pre-cutover testing with production data volumes in staging environments
• Cutover orchestration with go/no-go decision points
• Real-time validation during cutover (record counts, checksums, sampling)
• Reconciliation post-cutover (full data validation between source and target)
• Performance validation that target meets SLAs under production load

For high-stakes migrations, run table-top exercises and dress rehearsals before live cutover. The first time a team executes the cutover playbook should not be in production.

Stage 7: Stabilization and source decommission

Post-migration work:

• Application validation that downstream systems work correctly with target data
• Performance tuning based on production query patterns
• User acceptance confirmation across affected business teams
• Source system maintenance during transition period (often 30-90 days of parallel availability)
• Decommission of source systems after stability confirmed

Most migrations underestimate stabilization timeline. Plan 4-12 weeks post-cutover for full stabilization.

Five common failure modes

1. Inadequate discovery

Symptom: surprises mid-migration about data structures, volumes, or business dependencies.

Mitigation: invest in discovery upfront. The 4-8 weeks of audit work prevents months of remediation later.

2. Underestimating data quality work

Symptom: data prep takes 2-3x longer than planned.

Mitigation: profile actual data quality before scoping. Plan for 40-60% of project time on data preparation.

3. Big bang on complex systems

Symptom: migration fails partially, requiring complete rollback. Hours of business downtime turns into days.

Mitigation: use phased or parallel-run strategies for any migration touching critical business systems. Big bang only for small simple migrations.

4. Performance surprises in target

Symptom: queries that ran fast in source are slow in target. Reports that worked break under new schema.

Mitigation: load-test target under production query patterns before cutover. Tune target architecture for actual access patterns, not theoretical workloads.

5. Stakeholder coordination failures

Symptom: business teams discover broken workflows after cutover. Finger-pointing between IT and business owners.

Mitigation: detailed communication plan, business owner sign-off on cutover criteria, rollback authority defined clearly upfront.

Migration tooling we deploy

Modern data migration toolchain:

Discovery and inventory: Collibra, Alation, Atlan, AWS Glue Data Catalog.

ETL/ELT: Fivetran, Airbyte, dbt, Apache Airflow, Dagster.

Data quality: Great Expectations, Soda, Monte Carlo for ongoing observability.

Specialized migration tools: AWS DMS (Database Migration Service), Azure Database Migration Service, Google Database Migration Service, Striim for change data capture.

Cloud warehouses: Snowflake, BigQuery, Databricks, Redshift as common migration targets.

Reconciliation: Datafold, custom validation scripts, automated checksum comparison.

For most enterprise migrations, the toolchain combines: data catalog for discovery, modern ELT (Airbyte/Fivetran + dbt) for pipeline development, cloud warehouse as target, observability tools for ongoing monitoring.

Three migration project budgets

Small migration: Single database engine change, modest data volume, simple schema. $50K-$150K, 8-16 weeks.

Mid-size migration: Multiple source systems consolidation to cloud platform, moderate complexity. $200K-$600K, 16-32 weeks.

Enterprise platform migration: Legacy modernization, complex multi-source, regulatory compliance, organization-wide transition. $1M-$5M+, 9-18 months.

For broader cost framing, see our data analytics cost article.

Final framing

Data migrations succeed on planning discipline, not heroics. The teams that ship successful migrations invest in discovery upfront, design targets carefully, choose strategies that match risk tolerance, prepare data thoroughly, validate continuously, and coordinate stakeholders relentlessly.

The teams that skip these steps and rely on technical heroics during cutover overrun reliably. The technology doesn't determine success or failure — execution discipline does.

---

Ready to scope a data migration? Run the Project Estimator for a deterministic ballpark, or book a 45-minute Discovery with our data engineering team — we'll review your source systems, target architecture, and risk tolerance, and tell you honestly what migration approach fits your scope.