Remote Senior Data Engineer — Scalable Data Pipelines

540; Arlington (VA); On-site; USD 150,000 - 190,000; Full time

Benefits offered by this job

Flexible PTO

Federal holidays off

Health, dental and vision insurance

Flexible Spending Account (FSA)

401k with employer match

Life insurance, short- and long-term

Company events

Tickets to Washington Capitals & Nats

Job summary

540 is seeking a Senior Data Engineer to support a federal health data modernization initiative. You will design scalable data pipelines and APIs in a Databricks environment and guide teams on best practices.

You will own data ingestion, integration, and governance while collaborating with DoW and federal health stakeholders to deliver reusable data products and reliable analytics capabilities.

Qualifications

• Bachelor's degree in Computer Science, Engineering, Data Science, or a related technical field.
• 10+ years of data/ software engineering experience.
• Advanced proficiency with Python and SQL.
• Strong experience with Apache Spark and distributed data processing.
• Experience working with Databricks or similar modern data platforms.
• Experience designing and maintaining ETL/ELT for large-scale datasets.
• Experience integrating data across disparate systems and API-based data integrations.
• Strong understanding of data modeling, data architecture, data quality, and data governance.
• Experience troubleshooting production data pipelines for performance and reliability.
• Experience with Git-based development workflows and CLI.
• Ability to mentor engineers and guide engineering practices.

Responsibilities

• Design, develop, and maintain scalable, production-ready data pipelines and data products using Spark (Python/SQL) in a Databricks environment.
• Lead the integration and transformation of data from diverse DoW and federal health systems into reusable data products.
• Design scalable approaches for data ingestion, integration, and exchange, including API-based integrations.
• Establish and promote reusable data engineering patterns, standards, and best practices across data products.
• Provide technical guidance on data architecture, pipeline design, data modeling, integration approaches, and engineering practices.
• Monitor, troubleshoot, and optimize production workflows and data pipelines for performance and reliability.
• Define and implement data validation, quality, and governance practices to improve data usability.
• Troubleshoot complex data integration challenges, identify root causes, and drive sustainable solutions.
• Collaborate with engineers, architects, analysts, and stakeholders to translate complex data needs into scalable solutions.
• Provide mentorship to engineers and help teams navigate technical challenges.

Skills

Python

SQL

Spark

Databricks

ETL/ELT design

APIs

Data modeling

Data governance

Data ingestion

Mentoring engineers

Git workflows

CLI

Education

Bachelor’s Degree in Computer Science/Engineering/Data Science or related field

Tools

Databricks

Git workflows

Job description

540 is seeking a Senior Data Engineer to support a federal health data modernization initiative. You will design scalable data pipelines and APIs in a Databricks environment and guide teams on best practices.

You will own data ingestion, integration, and governance while collaborating with DoW and federal health stakeholders to deliver reusable data products and reliable analytics capabilities.