Data Engineer: Databricks Lakehouse for AI/ML workloads

540; Arlington (VA); On-site; USD 120,000 - 180,000; Full time

Benefits offered by this job

Flexible PTO

Health, dental and vision insurance

401k with employer match

Professional development

Paid cloud developer accounts

Referral bonuses

Office perks (parking, metro)

HQ events

Capitals/Nationals tickets

Job summary

540 is seeking a Data Engineer to support a mission-critical modernization effort for the Department of War. You will design, build, and maintain Databricks-based data pipelines and lakehouse capabilities at enterprise scale.

Working with software engineers, AI/ML teams, cybersecurity and mission stakeholders, you will create scalable, reliable solutions using Databricks, Python, Spark, and Delta Lake to enable secure data integration, analytics, and AI/ML workloads.

Qualifications

• 4+ years of relevant data engineering or software engineering experience.
• Hands-on experience developing and operating production data pipelines using Databricks.
• Proficiency with Python, SQL, PySpark, Apache Spark, and Delta Lake.
• Experience building automated ETL/ELT pipelines for large-scale datasets.
• Experience designing and maintaining data models, schemas, tables, and lakehouse architectures.
• Experience managing Databricks notebooks, jobs, workflows, and compute resources.
• Experience implementing data quality, automated testing, monitoring, lineage, or metadata-management capabilities.
• Experience working with Databricks and cloud-based data services in AWS, Azure, or Google Cloud.
• Experience with structured, semi-structured, and unstructured data.
• Understanding of lakehouse architecture, data governance, security, privacy, and access-control principles.
• Ability to troubleshoot data pipelines, Spark workloads, infrastructure, and applications.

Responsibilities

• Design, develop, and maintain Databricks-based data pipelines, data products, and lakehouse capabilities
• Build automated ETL/ELT pipelines that ingest, transform, and deliver mission-critical data
• Develop production-grade data-processing solutions using Python, SQL, PySpark, Apache Spark, and Delta Lake
• Design and maintain data models, schemas, tables, and medallion architecture patterns supporting analytical, operational, and AI/ML workloads
• Build and operate batch and streaming data pipelines supporting mission requirements
• Develop and manage Databricks notebooks, jobs, workflows, clusters, and compute resources
• Implement data-quality checks, automated testing, monitoring, lineage, and metadata-management capabilities
• Support data discovery, governance, and access controls using Unity Catalog or similar technologies
• Optimize Spark workloads and Databricks resources for performance, scalability, reliability, and cost efficiency
• Collaborate with engineers, analysts, and data scientists to deliver reusable data products and mission capabilities
• Support Databricks deployments using CI/CD, infrastructure as code, and source control
• Partner with cybersecurity teams to implement data-protection, access-control, auditing, and governance requirements
• Troubleshoot issues affecting Databricks workloads, data pipelines, storage systems, and production data services
• Document data models, pipeline designs, engineering processes, and operational procedures

Skills

Databricks

Python

SQL

PySpark

Apache Spark

Delta Lake

Data Pipelines

Lakehouse

CI/CD

Cloud services

Education

Bachelor's degree in Computer Science, Engineering, or related field

Tools

Databricks notebooks

Airflow

Kubernetes

Docker

Unity Catalog

Job description

540 is seeking a Data Engineer to support a mission-critical modernization effort for the Department of War. You will design, build, and maintain Databricks-based data pipelines and lakehouse capabilities at enterprise scale.

Working with software engineers, AI/ML teams, cybersecurity and mission stakeholders, you will create scalable, reliable solutions using Databricks, Python, Spark, and Delta Lake to enable secure data integration, analytics, and AI/ML workloads.