Data Engineer (TS/SCI) – Databricks & DoD Pipelines

NextGenEnergyJobs; Arlington, Northern (VA, KY); Hybrid; USD 120,000 - 160,000; Full time

Job summary

Redhorse Corporation is seeking a data engineer to ingest, transform, and optimize large datasets across DoD environments. You will build scalable ETL/ELT pipelines using Databricks (Delta Lake, Lakehouse) and PySpark, integrate with Qlik Sense/View BI, and ensure security compliance across classified networks.

Candidates must hold TS clearance with SCI eligibility and have experience in Advana Data Environment and CI/CD tools such as GitLab/Jira/Jenkins.

Qualifications

• Hands-on experience developing data workflows using Databricks (e.g., Delta Lake, Lakehouse architecture, PySpark).
• Experience integrating data pipelines with business intelligence and data visualization platforms, particularly Qlik Sense or QlikView.
• Experience working within the DoD Advana Data Environment.
• Familiarity with deploying and engineering data pipelines across classified networks and multiple security domains.
• Experience with CI/CD tools and version control systems such as GitLab, Jira, or Jenkins.
• Active Top Secret (TS) clearance with Sensitive Compartmented Information (SCI) eligibility.

Responsibilities

• Ingest, configure, and transform structured, semi-structured, and unstructured datasets to meet mission partner and stakeholder requirements.
• Build, automate, and maintain scalable ETL/ELT pipelines supporting DevOps and rapid analytics delivery.
• Maintain data pipeline operations, monitor environment health, and implement automated performance optimization and cost-reduction practices.
• Work under general guidance to prototype new data capabilities, evaluate emerging technologies, and deploy end-to-end data solutions.
• Collaborate with cross-functional technical teams to troubleshoot pipeline bottlenecks, optimize database structures, and implement secure data retrieval methods.
• Implement and automate DoD Security Technical Implementation Guides (STIGs) across data systems to ensure strict compliance and system security.

Skills

Databricks

Delta Lake

PySpark

Qlik Sense

QlikView

Advana DoD

CI/CD

GitLab

Jira

Jenkins

TS/SCI clearance

Job description

Redhorse Corporation is seeking a data engineer to ingest, transform, and optimize large datasets across DoD environments. You will build scalable ETL/ELT pipelines using Databricks (Delta Lake, Lakehouse) and PySpark, integrate with Qlik Sense/View BI, and ensure security compliance across classified networks.

Candidates must hold TS clearance with SCI eligibility and have experience in Advana Data Environment and CI/CD tools such as GitLab/Jira/Jenkins.