Senior Databricks Lakehouse Migration Engineer

ECS Corporate Services; Arlington (VA); Hybrid; USD 150,000 - 160,000; Full time

Job summary

ECS Corporate Services seeks a Senior Databricks Migration Engineer to lead a hybrid Arlington-based data migration. You will move from legacy SQL Server/ADF to Databricks Lakehouse, build scalable ETL pipelines, and optimize for Power BI performance while enforcing governance and security standards.

The role requires 5+ years in data engineering, cloud platforms, and leadership of cross-functional data projects; you will design robust data architectures, document patterns, and enable the team

Qualifications

• 5+ years' experience in data engineering, data system development or related roles.
• 5+ years' experience with cloud platforms (e.g. Azure, AWS, GCP).
• 1+ year leading complex, cross-functional data projects and technical teams.
• Experience with Databricks Lakehouse, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms.
• Experience with data warehousing, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark.
• Mastery of data engineering principles, including data modeling, ETL (Extract, Transform, Load) processes, and data pipelines.
• Proficiency with Azure Data Lake data storage and processing services.
• Skilled at designing, building, and optimizing data pipelines for ingesting, transforming, and loading data.
• Proficiency in SQL and Python/PySpark for data manipulation and pipeline development.
• Skilled at identifying and resolving data-related challenges.
• Skilled at creating efficient data models that meet business requirements.
• Skilled at optimizing query performance and system scalability.

Responsibilities

• Lead the technical migration from legacy SQL Server stored procedures and ADF pipelines to Databricks Lakehouse (Delta Lake).
• Translate relational data warehousing paradigms into scalable Lakehouse frameworks (Bronze, Silver, Gold).
• Design robust ETL/ELT frameworks using PySpark, Delta Live Tables (DLT), and Databricks Workflows.
• Architect Gold Layer to maximize Power BI performance.
• Optimize Databricks SQL Warehouses for high-concurrency Power BI queries (DirectQuery/Import).
• Implement optimization techniques: Z-Ordering, data skipping, liquid clustering, materialized views.
• Define governance for cluster sizing, auto-scaling policies, serverless SQL compute for cost/performance balance.
• Build monitoring dashboards for DBU consumption and cost savings.
• Establish partition strategies and file size management within Delta Lake.
• Design and implement a robust data security model using Unity Catalog with RBAC.
• Enforce row/column security for Power BI consumers and internal analysts.
• Align Lakehouse security with Azure AD and RBAC standards.
• Lead pair-programming sessions, workshops, and code reviews; create technical docs and playbooks.
• Develop knowledge transfer to ensure team self-sufficiency post-migration.
• Communicate effectively to technical and non-technical audiences.

Skills

Data engineering

Cloud platforms

Databricks Lakehouse

Apache Spark

Delta Lake

PySpark

SQL

Python

Data pipelines

ETL/ELT

Data modeling

Dimensional modeling

Power BI

Azure Data Lake

Education

Bachelor's degree in Computer Science, Engineering, or related field

Tools

Databricks Lakehouse

Apache Spark

Delta Lake

Azure Data Lake

Power BI

Job description

ECS Corporate Services seeks a Senior Databricks Migration Engineer to lead a hybrid Arlington-based data migration. You will move from legacy SQL Server/ADF to Databricks Lakehouse, build scalable ETL pipelines, and optimize for Power BI performance while enforcing governance and security standards.

The role requires 5+ years in data engineering, cloud platforms, and leadership of cross-functional data projects; you will design robust data architectures, document patterns, and enable the team