Senior Data Engineer

540; Arlington (VA); On-site; USD 160,000 - 260,000; Full time

Benefits offered by this job

Flexible PTO + federal holidays off

Health, dental and vision insurance

401k with employer match

Professional development (training, &

Paid cloud developer accounts

HQ office perks (parking / metro)

Job summary

540 is seeking a Senior Data Engineer in Arlington, VA to lead the design and evolution of Databricks-based data pipelines and lakehouse capabilities, enabling secure data integration, analytics, AI/ML, and operational workloads at enterprise scale.

You will collaborate with engineers, architects, cybersecurity teams, and mission stakeholders to translate requirements into scalable, secure data architectures using Databricks, Spark, and Delta Lake, while defining standards and mentoring

Qualifications

• 9+ years of relevant data engineering or software engineering experience.
• Experience leading enterprise-scale data platform and pipeline implementations using Databricks.
• Advanced proficiency with Python, SQL, PySpark, Apache Spark, and Delta Lake.
• Experience architecting large-scale ETL/ELT, batch, and streaming pipelines.
• Experience designing lakehouse architectures, data models, schemas, and data contracts.
• Experience managing and optimizing Databricks jobs, workflows, compute resources, and Spark workloads.
• Experience implementing data quality, monitoring, lineage, metadata management, and governance capabilities.
• Experience with Unity Catalog or similar data-governance and access-control solutions.
• Experience operating Databricks within AWS, Azure, or Google Cloud.
• Experience with CI/CD, infrastructure as code, automated testing, and source control.
• Strong understanding of data security, privacy, governance, and access-control principles.
• Experience leading technical reviews, mentoring engineers, and communicating architecture decisions.

Responsibilities

• Lead the architecture and evolution of Databricks-based data pipelines and lakehouse capabilities.
• Translate mission requirements into scalable data architectures and implementation strategies.
• Define data engineering standards, reusable patterns, and best practices across engineering teams.
• Architect automated ETL/ELT pipelines using Python, SQL, PySpark, Apache Spark, and Delta Lake.
• Design scalable data models, schemas, data contracts, and medallion architecture patterns.
• Lead the development of batch and streaming capabilities supporting operational, analytical, and AI/ML workloads.
• Establish data quality, lineage, metadata, observability, and governance practices using Unity Catalog or similar technologies.
• Optimize Databricks and Spark workloads for performance, scalability, reliability, and cost efficiency.
• Establish CI/CD, infrastructure-as-code, testing, monitoring, and operational practices for Databricks environments.
• Lead design reviews and resolve complex issues spanning data pipelines, infrastructure, and production services.
• Partner with cybersecurity teams to incorporate security, access control, auditing, and governance requirements.
• Communicate architecture decisions and mentor engineers on Databricks and data engineering best practices.

Skills

Databricks

Python

SQL

PySpark

Apache Spark

Delta Lake

ETL/ELT design

Data architecture

CI/CD in data

Security & governance

Education

Bachelor's degree in Computer Science/Engineering

Tools

Unity Catalog

Terraform

Airflow

Databricks

Job description

540 is seeking a Senior Data Engineer to support a mission-critical technology modernization effort for the Department of War. You will lead the design and evolution of Databricks-based data pipelines and lakehouse capabilities that enable secure data integration, analytics, AI/ML, and operational workloads at enterprise scale.

Working with engineers, architects, cybersecurity teams, and mission stakeholders, you will translate complex requirements into secure, scalable solutions using Databricks, Apache Spark, and Delta Lake. You will define engineering standards, guide technical delivery, and mentor engineers while ensuring data quality, governance, and platform reliability.

Location

Arlington, VA

Citizenship & Clearance Requirement

Per client requirements, candidates must be U.S. Citizens with an active DoW Secret (or higher) clearance

Education Requirement

Bachelor's degree in Computer Science, Engineering, or a related technical field preferred; equivalent combinations of education and relevant experience will be considered

540 Internal Thrive Level

Senior Data Engineer

WHY 540?

540 is a forward-thinking company that the government turns to in order to #getshitdone. We don't just talk about innovation – we deliver it. We break down barriers, build impactful technology, and solve mission-critical problems.

HOW YOU'LL DRIVE IMPACT

• Lead the architecture and evolution of Databricks-based data pipelines and lakehouse capabilities
• Translate mission requirements into scalable data architectures and implementation strategies
• Define data engineering standards, reusable patterns, and best practices across engineering teams
• Architect automated ETL/ELT pipelines using Python, SQL, PySpark, Apache Spark, and Delta Lake
• Design scalable data models, schemas, data contracts, and medallion architecture patterns
• Lead the development of batch and streaming capabilities supporting operational, analytical, and AI/ML workloads
• Establish data quality, lineage, metadata, observability, and governance practices using Unity Catalog or similar technologies
• Optimize Databricks and Spark workloads for performance, scalability, reliability, and cost efficiency
• Establish CI/CD, infrastructure-as-code, testing, monitoring, and operational practices for Databricks environments
• Lead design reviews and resolve complex issues spanning data pipelines, infrastructure, and production services
• Partner with cybersecurity teams to incorporate security, access control, auditing, and governance requirements
• Communicate architecture decisions and mentor engineers on Databricks and data engineering best practices

REQUIRED SKILLS & EXPERIENCE

• 9+ years of relevant data engineering or software engineering experience
• Experience leading enterprise-scale data platform and pipeline implementations using Databricks
• Advanced proficiency with Python, SQL, PySpark, Apache Spark, and Delta Lake
• Experience architecting large-scale ETL/ELT, batch, and streaming pipelines
• Experience designing lakehouse architectures, data models, schemas, and data contracts
• Experience managing and optimizing Databricks jobs, workflows, compute resources, and Spark workloads
• Experience implementing data quality, monitoring, lineage, metadata management, and governance capabilities
• Experience with Unity Catalog or similar data-governance and access-control solutions
• Experience operating Databricks within AWS, Azure, or Google Cloud
• Experience with CI/CD, infrastructure as code, automated testing, and source control
• Strong understanding of data security, privacy, governance, and access-control principles
• Experience leading technical reviews, mentoring engineers, and communicating architecture decisions

NICE TO HAVE

• Relevant Databricks certification
• Experience supporting DoW, federal, Advana, or other mission data environments
• Experience building data platforms in classified, regulated, or mission-critical environments
• Experience with Terraform, Databricks Asset Bundles, Airflow, or similar automation and orchestration tools
• Experience architecting streaming solutions using Kafka, Kinesis, Pulsar, or Spark Structured Streaming
• Experience supporting AI/ML pipelines, feature platforms, or MLflow
• Currently holds, or is willing to obtain within 30 days of employment, an approved certification such as Cloud+, GSEC, Security+, or SSCP

BENEFITS & PERKS

• Flexible PTO + all Federal holidays off
• Health, dental and vision insurance plans
• Flexible Spending Account (FSA)
• 401k with employer match
• Company-sponsored life insurance, short- and long-term disability
• Professional development (training, certifications, conferences)
• Paid cloud developer accounts
• Referral bonuses
• HQ office perks (parking / metro reimbursement, nitro coffee & lunches)
• Annual social events (540 Week, hackathon, charity golf tournament, etc.)
• Access to 540's Washington Capitals & Nationals tickets

EQUAL EMPLOYMENT OPPORTUNITY (EEO)

540's policy is to provide equal employment opportunity to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.

This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation and training.