Senior Data Engineer for Health Data Pipelines (Databricks)

Systems Planning & Analysis; United States; On-site; USD 156,000 - 179,000; Full time

Job summary

Intrepid, an SPA Company, is seeking a Senior Data Engineer to build scalable data pipelines and data products for the Defense Health Agency Chief Data and Analytics Office. You will integrate data from diverse defense and federal health systems to support enterprise data orchestration and API-based data exchanges.

The role requires 10+ years in data or software engineering, proficiency with Python/SQL and Spark, and experience with Databricks and ETL/ELT processes.

Qualifications

• Master’s degree in Computer Science, Information Systems, Software Engineering, or a related field.
• 10+ years of data engineering, software engineering, or related technical experience.
• Extensive hands-on experience designing, building, and operating production data pipelines and data products.
• Advanced proficiency with Python and SQL.
• Strong experience with Apache Spark and distributed data processing.
• Experience working with Databricks or similar modern data platforms.
• Experience designing and maintaining ETL/ELT processes for complex, large-scale datasets.
• Experience troubleshooting and optimizing complex production data pipelines for performance, reliability, and scalability.
• Experience working with Git-based development workflows and modern software engineering practices.
• Strong client and stakeholder communication skills.

Responsibilities

• Design, develop, and maintain scalable, production-ready data pipelines and data products using Spark (Python/SQL) in a Databricks environment.
• Lead the integration and transformation of complex data from diverse DoW and federal health systems and sources into reliable, reusable data products.
• Design scalable approaches for data ingestion, integration, and exchange, including API-based integrations and services.
• Establish and promote reusable data engineering patterns, standards, and best practices that improve consistency, scalability, and maintainability across data products.
• Provide technical guidance on data architecture, pipeline design, data modeling, integration approaches, and engineering practices.
• Monitor, troubleshoot, and optimize production workflows and data pipelines, identifying performance, reliability, and scalability improvements.
• Define and implement data validation, quality, and governance practices that improve the reliability and usability of data products.
• Troubleshoot complex technical and data integration challenges, identify root causes, and drive sustainable solutions.
• Collaborate with engineers, architects, analysts, and customer stakeholders to translate complex data needs into scalable technical solutions.
• Provide technical guidance and mentorship to other engineers, helping teams navigate complex or unfamiliar technical challenges.
• Proactively identify opportunities to improve engineering tools, processes, and patterns and help drive their adoption across the team.
• Take ownership of complex technical areas and help maintain engineering quality, consistency, and cohesion as the platform and portfolio of data products grow.

Skills

Python

SQL

Apache Spark

Databricks

ETL/ELT

Git workflows

Security/compliance awareness

Communication skills

Education

Master’s degree in Computer Science/Information Systems/Software Engineering or related field

Tools

Databricks

Git

AWS

Job description

Intrepid, an SPA Company, is seeking a Senior Data Engineer to build scalable data pipelines and data products for the Defense Health Agency Chief Data and Analytics Office. You will integrate data from diverse defense and federal health systems to support enterprise data orchestration and API-based data exchanges.

The role requires 10+ years in data or software engineering, proficiency with Python/SQL and Spark, and experience with Databricks and ETL/ELT processes.