Lead Data Engineer - Databricks Lakehouse Migration

Insight Global Technologies; United States; Remote; USD 120,000 - 155,000; Full time

Job summary

Insight Global Technologies is seeking a Data Engineer to lead a migration from AWS Glue/Redshift to Databricks in the United States. You will own the end-to-end migration roadmap, inventory pipelines, define wave-based plans, and architect the Databricks Lakehouse with Unity Catalog and cost guardrails.

You will re-platform pipelines using PySpark, Spark SQL, Delta Live Tables, and Databricks Workflows, establish data parity and regression testing, and drive cutover for BI tools, reverse ETL,

Qualifications

• 8+ years of professional data engineering experience with at least 2+ years in a tech-lead or lead engineer capacity.
• Deep hands-on experience with Apache Spark (PySpark and Spark SQL) and performance tuning.
• Hands-on production experience with Databricks Delta Lake, Unity Catalog, Workflows, and Delta Live Tables or similar pipelines.
• Strong production experience with AWS Glue (jobs, crawlers, Data Catalog).

Responsibilities

• Own the end-to-end migration roadmap: discovery, assessment, design, build, test, cutover, and decommissioning of legacy AWS Glue jobs, Redshift clusters, and tooling.
• Inventory and prioritize existing pipelines, stored procedures, materialized views, and Redshift workloads; define a wave-based migration plan with success criteria for each wave.
• Architect the target Databricks Lakehouse: topology, Unity Catalog, medallion layering, S3 storage, cluster and SQL Warehouse strategy, and cost guardrails.
• Re-platform pipelines from Glue/Redshift into Databricks using PySpark, Spark SQL, Delta Live Tables, or Databricks Workflows; refactor for performance or cost.
• Define data parity, reconciliation, and regression-testing strategy between legacy Redshift and the Lakehouse during dual-run periods.
• Plan and execute cutover for downstream consumers (BI tools, reverse ETL, ML feature stores, app reads); minimize disruption.
• Establish engineering standards for the new platform: repository structure, CI/CD, environment promotion, code review, and testing.
• Define and roll out data quality, observability, and lineage practices (Great Expectations, Unity Catalog lineage, monitoring).
• Govern access, security, and compliance via Unity Catalog: catalogs, schemas, row/column-level security, audit."] ,
• CoT_job_summary_short
• COMPANY NAME: Insight Global TechnologiesKEY POINTS: Migration from Glue/Redshift to Databricks, end-to-end roadmap ownership, databricks lakehouse design, cost optimization
• job_summary_short
• <p>Insight Global Technologies is seeking a Data Engineer to lead a migration from AWS Glue/Redshift to Databricks in the United States. You will own the end-to-end migration roadmap, inventory pipelines, define wave-based plans, and architect the Databricks Lakehouse with Unity Catalog and cost guardrails.</p><p>You will re-platform pipelines using PySpark, Spark SQL, Delta Live Tables, and Databricks Workflows, establish data parity and regression testing, and drive cutover for BI tools, reverse ETL, and ML features while setting engineering standards for the new platform.</p>
• contract_type
• fulltime
• location_type
• on site
• remote_scope
• null
• perks
• []
• stated_min
• null
• stated_max
• null
• stated_absolute
• null
• salary_estimation_cot
• Senior Data Engineer in United States: market range 120000-155000 USD per year based on experience and location.
• estimated_low
• 120000
• estimated_medium
• 135000
• estimated_high
• 155000
• currency
• USD
• frequency
• yearly
• bonus
• null
• tax
• gross
• application_email
• null
• job_end_date
• null
• contact_person
• null
• hiring_department_name
• null

Skills

Apache Spark

PySpark

Spark SQL

Databricks Delta Lake

Unity Catalog

Databricks Workflows

Delta Live Tables

AWS Glue

Performance tuning

Leadership

Tools

GitHub Actions

Azure DevOps

Terraform

Delta Live Tables

Job description

Insight Global Technologies is seeking a Data Engineer to lead a migration from AWS Glue/Redshift to Databricks in the United States. You will own the end-to-end migration roadmap, inventory pipelines, define wave-based plans, and architect the Databricks Lakehouse with Unity Catalog and cost guardrails.

You will re-platform pipelines using PySpark, Spark SQL, Delta Live Tables, and Databricks Workflows, establish data parity and regression testing, and drive cutover for BI tools, reverse ETL,