Senior Databricks Migration Engineer

ECS; Arlington (VA); On-site; USD 130,000 - 158,000; Full time

Job summary

ECS Federal LLC seeks a Senior Databricks Migration Engineer to lead data migration from legacy systems to Databricks Lakehouse, designing Bronze-Silver-Gold pipelines and robust ETL/ELT frameworks. You will optimize for Power BI performance and implement governance, security, and cost controls across the Lakehouse stack.

You will collaborate with cross-functional teams, mentor on Spark-based workloads, and produce architecture docs and playbooks to ensure a smooth transition for the USPS

Qualifications

• Bachelor's degree or higher in Computer Science, Engineering, or related field.
• 5+ years of data engineering or related roles.
• 5+ years on cloud platforms (Azure, AWS, GCP).
• 1+ year leading complex cross-functional data projects and teams.
• Experience with Databricks Lakehouse, Spark, Delta Lake, and distributed compute platforms.

Responsibilities

• Lead migration from legacy SQL Server and ADF pipelines to Databricks Lakehouse with best-practice design.
• Translate relational warehousing concepts to scalable Lakehouse architectures (Bronze, Silver, Gold).
• Design ETL/ELT frameworks using PySpark, Delta Live Tables, and Databricks Workflows.
• Architect Gold Layer models to optimize Power BI performance.
• Optimize Databricks SQL Warehouses for high-concurrency Power BI queries.
• Define governance for cluster sizing, auto-scaling, and serverless SQL compute.
• Create monitoring dashboards for DBU usage and cost-saving opportunities.
• Develop robust partition strategies and Delta Lake file size guidance.
• Build data security model with Unity Catalog and row/column-level access.

Skills

Databricks Lakehouse

Apache Spark

Delta Lake

Python/PySpark

SQL

Education

Bachelor's degree or higher (CS/Engineering)

Tools

Delta Live Tables (DLT)

Databricks Workflows

Azure AD RBAC

Unity Catalog

Job description

Senior Databricks Migration Engineer

Location: Arlington, VA office for US Postal Services' Inspector General's Office (OIG). This role leads technical data migration efforts from legacy SQL systems, warehouses, and servers to Databricks Lakehouse (Delta Lake), ensuring best practice Lakehouse design. It also involves building large data structures, dashboards, and reporting on data migration projects.

Responsibilities

• Lead the technical migration from legacy SQL Server stored procedures and ADF pipelines to Databricks Lakehouse (Delta Lake), ensuring best practice Lakehouse design.
• Translate traditional relational data warehousing paradigms into scalable, distributed Lakehouse frameworks (Bronze, Silver, Gold).
• Design robust, reusable ETL/ELT frameworks using PySpark, Delta Live Tables (DLT), and Databricks Workflows.
• Architect and refine the Gold Layer (dimensional models, star schemas) specifically to maximize Power BI performance.
• Optimize Databricks SQL Warehouses to support high-concurrency, low-latency Power BI queries (DirectQuery and Import modes).
• Implement advanced optimization techniques, including Z-Ordering, data skipping, liquid clustering, and materialized views.
• Define and enforce governance standards for cluster sizing, auto-scaling policies, and serverless SQL compute to balance performance with cost.
• Implement proactive monitoring dashboards to track Databricks Unit (DBU) consumption and identify cost-saving opportunities.
• Establish best practices for partition strategies and file size management within Delta Lake.
• Design and implement a robust data security model using Unity Catalog for centralized governance.
• Enforce row-level and column-level security policies to ensure compliant data access for Power BI consumers and internal analysts.
• Align the Lakehouse security architecture with existing enterprise Azure Active Directory (Microsoft Entra ID) and RBAC standards.
• Act as the primary technical lead, conducting dedicated pair-programming sessions, workshops, and code reviews to transition the team from SQL-centric to Spark-centric thinking.
• Create comprehensive technical documentation, including architecture diagrams, design patterns, and optimization playbooks.
• Build a foundational knowledge transfer framework to ensure the internal team is fully self-sufficient post-migration.
• Communicate effectively verbally and in written form to both technical and non-technical audience.
• Work in an organized fashion, completing tasks timely while paying close attention to details.

Required Skills

• Bachelor's degree or higher from an accredited college or university in Computer Science, Engineering, or a related technical field.
• 5+ years’ experience in data engineering, data system development or related roles.
• 5+ years’ experience with cloud platforms (e.g. Azure, AWS, GCP).
• 1+ year leading complex, cross-functional data projects and technical teams.
• Experience with Databricks Lakehouse, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms.
• Experience with data warehousing, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark.

Desired Skills

• Mastery of data engineering principles, including data modeling, ETL (Extract, Transform, Load) processes, and data pipelines.
• Proficiency with Azure Data Lake data storage and processing services.
• Skilled at designing, building, and optimizing data pipelines for ingesting, transforming, and loading data.
• Proficiency in languages such as SQL and Python/PySpark for data manipulation and pipeline development.
• Skilled at identifying and resolving data-related challenges.
• Skilled at creating efficient data models that meet business requirements.
• Skilled at optimizing query performance and system scalability.

Additional Requirements

• This position is contingent upon additional funding and requires a Public Trust background investigation, including an in-depth background check and either US Citizenship or Permanent Resident (Green Card) status.
• Requires working onsite 4 days a week at the USPS customer site in Arlington, VA.

Salary Range: $130,000 - $158,000 annually.

ECS Federal LLC is an equal opportunity employer and does not discriminate or allow discrimination on the basis of any characteristic protected by law. All qualified applicants will receive consideration for employment without regard to disability, status as a protected veteran or any other status protected by applicable federal, state, or local jurisdiction law.