Senior Databricks Lakehouse Migration Lead

MDW Associates; Arlington (VA); On-site; USD 140,000 - 190,000; Full time

Job summary

MDW Associates is seeking a Senior Databricks Migration Engineer to lead data migration from legacy SQL systems to Databricks Lakehouse (Delta Lake) and build scalable ETL/ELT pipelines. The role focuses on architecture, governance, and performance tuning for Power BI reporting.

The position is on-site 4 days a week at the USPS customer site in Arlington, VA, requiring hands-on leadership, pair programming, and strong communication across technical and non-technical teams.

Qualifications

• 5+ years' experience in data engineering or related roles.
• 5+ years' experience with cloud platforms (Azure, AWS, GCP).
• 1+ year leading complex, cross-functional data projects.
• Experience with Databricks Lakehouse, Spark, Delta Lake, and distributed compute.
• Experience with data warehousing, dimensional modeling, and metadata-driven ingestion.

Responsibilities

• Lead migration from legacy SQL Server to Databricks Lakehouse (Delta Lake).
• Design scalable Lakehouse architectures and robust ETL/ELT frameworks using PySpark and DLT.
• Architect Gold layer models (dimensional, star schemas) for Power BI performance.
• Optimize Databricks SQL Warehouses for high concurrency and low latency Power BI queries.
• Govern cluster sizing, auto-scaling, and serverless SQL compute with cost control.
• Establish security via Unity Catalog and RBAC, including row- and column-level access control.
• Create governance, monitoring dashboards, and comprehensive documentation.
• Coordinate with Azure AD and enterprise security standards; lead knowledge transfer.

Skills

Databricks Lakehouse

Apache Spark

Delta Lake

PySpark

Power BI

SQL

Azure

Governance

Education

Bachelor's degree in Computer Science or related

Tools

Databricks

Unity Catalog

DLT (Delta Live Tables)

Job description

MDW Associates is seeking a Senior Databricks Migration Engineer to lead data migration from legacy SQL systems to Databricks Lakehouse (Delta Lake) and build scalable ETL/ELT pipelines. The role focuses on architecture, governance, and performance tuning for Power BI reporting.

The position is on-site 4 days a week at the USPS customer site in Arlington, VA, requiring hands-on leadership, pair programming, and strong communication across technical and non-technical teams.