Senior Data Engineer – DoD Secret Data & Streaming

OMNI; Texas; On-site; USD 120,000 - 180,000; Full time

Job summary

Omni Federal is seeking a Senior Data Engineer to design and operate modern data pipelines for Air Force programs. You will build streaming and batch workflows, ensure data quality, and enable discovery and governance across a lakehouse and data mesh.

Work involves Terraform, AWS services, Elastic search of metadata, and collaboration in an agile team to deliver secure, scalable data solutions for government clients.

Qualifications

• 8+ years of professional data engineering experience with production pipelines.
• Hands-on design and operation of streaming and batch data pipelines (Debezium, MSK, Spark, Airflow).
• Strong proficiency in Python and SQL with multi-source data validation.
• Experience with AWS data services and containerized environments delivered through DevSecOps/CI-CD pipelines.

Responsibilities

• Design, build, and operate streaming (Debezium into Amazon MSK) and batch (Spark on AWS EMR) pipelines.
• Validate data quality, lineage, and governance before exposing data to downstream users.
• Maintain metadata/catalogs (AWS Glue) and API Gateway integrations for data discovery.
• Contribute to an agile Kanban/Scrum team, participate in design reviews and code reviews.
• Manage infrastructure as code (Terraform) and secure deployment in line with DoD standards.

Skills

Streaming data pipelines

Debezium

Amazon MSK

Apache Spark

AWS EMR

Python

SQL

Terraform

Kubernetes

Data governance

Tools

Debezium

Amazon MSK

Apache Spark

Apache Airflow

Terraform

AWS Glue

AWS S3

Apache Iceberg

Kubernetes

AWS Aurora

Job description

Omni Federal is seeking a Senior Data Engineer to design and operate modern data pipelines for Air Force programs. You will build streaming and batch workflows, ensure data quality, and enable discovery and governance across a lakehouse and data mesh.

Work involves Terraform, AWS services, Elastic search of metadata, and collaboration in an agile team to deliver secure, scalable data solutions for government clients.