Senior Databricks Migration Engineer

PLANIT Group; Reston (VA); On-site; USD 130,000 - 170,000; Full time

Job summary

PLANIT Group seeks a Senior Databricks Migration Engineer to lead a government-focused migration project in Reston, VA, with flex to remote work. The role requires transitioning from SQL Server/ADF to Databricks Lakehouse, implementing Delta Lake, DLT, and rigorous governance while enabling Power BI performance.

You will guide a cross-functional team, design scalable data architectures, and deliver comprehensive documentation and knowledge transfer to ensure long-term operational

Qualifications

• Bachelor’s degree or higher in Computer Science, Engineering, or related technical field
• 5+ years in data engineering, data systems, or related roles
• 5+ years on cloud platforms (Azure, AWS, GCP)
• 1+ year leading complex, cross-functional data projects and teams

Responsibilities

• Lead migration from legacy SQL Server and ADF to Databricks Lakehouse (Delta Lake) with best-practice design
• Translate relational warehousing to scalable Lakehouse architectures (Bronze/Silver/Gold)
• Design ETL/ELT frameworks using PySpark, DLT, and Databricks Workflows
• Refine Gold Layer for Power BI performance (dimensional models, star schemas)
• Optimize Databricks SQL Warehouses for high-concurrency Power BI queries
• Implement advanced optimization: Z-Ordering, data skipping, liquid clustering, materialized views
• Enforce governance: cluster sizing, auto-scaling, serverless SQL compute
• Create monitoring dashboards to track DBU consumption and costs
• Establish partitioning strategies and Delta Lake file-size management
• Design data security model with Unity Catalog and access controls
• Align with Azure AD/Microsoft Entra ID and RBAC standards
• Lead pair-programming sessions and code reviews to shift from SQL-centric to Spark-centric
• Produce architecture diagrams, patterns, and optimization playbooks
• Deliver knowledge transfer to ensure self-sufficiency post-migration
• Communicate clearly with technical and non-technical audiences
• Maintain organized, detail-oriented workflow

Skills

Databricks Lakehouse

PySpark

Delta Live Tables

Databricks Workflows

Power BI

Delta Lake

RBAC

Azure Active Directory

Security governance

DBU monitoring

Partition strategies

Unity Catalog

Education

Bachelor's degree in Computer Science / Engineering or related field

Tools

Databricks

Delta Lake

Delta Live Tables (DLT)

Power BI

Job description

Reston, VA Contract On-Site Flexibility/Remote: 100% Senior Databricks Migration Engineer

We are seeking a Senior Databricks Migration Engineer to join our team of qualified, diverse individuals supporting a Federal Government customer in Rosslyn, VA. Candidates MUST be US citizens.

The individual serves as the authoritative resource for the agency who specializes in preparing big data infrastructure for analytical or operational uses. They are responsible for designing and creating systems that collect, manage, and convert raw data into usable information for data scientists and business analysts to interpret and enables the agency to make smarter decisions and optimize operations.

Position Requirements

Primary Requirements

• Lead the technical migration from legacy SQL Server stored procedures and ADF pipelines to Databricks Lakehouse (Delta Lake), ensuring best practice Lakehouse design.
• Translate traditional relational data warehousing paradigms into scalable, distributed Lakehouse frameworks (Bronze, Silver, Gold).
• Design robust, reusable ETL/ELT frameworks using PySpark, Delta Live Tables (DLT), and Databricks Workflows.
• Architect and refine the Gold Layer (dimensional models, star schemas) specifically to maximize Power BI performance.
• Optimize Databricks SQL Warehouses to support high-concurrency, low-latency Power BI queries (DirectQuery and Import modes).
• Implement advanced optimization techniques, including Z-Ordering, data skipping, liquid clustering, and materialized views.
• Define and enforce governance standards for cluster sizing, auto-scaling policies, and serverless SQL compute to balance performance with cost.
• Implement proactive monitoring dashboards to track Databricks Unit (DBU) consumption and identify cost-saving opportunities.
• Establish best practices for partition strategies and file size management within Delta Lake.
• Design and implement a robust data security model using Unity Catalog for centralized governance.
• Enforce row-level and column-level security policies to ensure compliant data access for Power BI consumers and internal analysts.
• Align the Lakehouse security architecture with existing enterprise Azure Active Directory (Microsoft Entra ID) and RBAC standards.
• Act as the primary technical lead, conducting dedicated pair-programming sessions, workshops, and code reviews to transition the team from SQL-centric to Spark-centric thinking.
• Create comprehensive technical documentation, including architecture diagrams, design patterns, and optimization playbooks.
• Build a foundational knowledge transfer framework to ensure the internal team is fully self-sufficient post-migration.
• Communicate effectively verbally and in written form to both technical and non-technical audience.
• Work in an organized fashion, completing tasks timely while paying close attention to details.

Desired KSAs

• Mastery of data engineering principles, including data modeling, ETL (Extract, Transform, Load) processes, and data pipelines
• Proficiency with Azure Data Lake data storage and processing services
• Skilled at designing, building, and optimizing data pipelines for ingesting, transforming, and loading data
• Proficiency in languages such as SQL and Python/PySpark for data manipulation and pipeline development
• Skilled at identifying and resolving data-related challenges
• Skilled at creating efficient data models that meet business requirements.
• Skilled at optimizing query performance and system scalability

Experience

• Must have a Bachelor's degree or higher from an accredited college or university in Computer Science, Engineering, or a related technical field
• 5+ years' experience in data engineering, data system development or related roles
• 5+ years' experience with cloud platforms (e.g. Azure, AWS, GCP)
• 1+ year leading complex, cross-functional data projects and technical teams
• Experience with Databricks Lakehouse, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms
• Experience with data warehousing, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark
• Candidates MUST be a US Citizen and be able to obtain a Position of Public Trust Clearance. Must not have traveled outside the US for a combined total of 6 months or more in last 5 years and have resided in the US for the last 5 years