Senior Data Engineer

OMNI; Texas; On-site; USD 120,000 - 180,000; Full time

Job summary

Omni Federal is seeking a Senior Data Engineer to design and operate modern data pipelines for Air Force programs. You will build streaming and batch workflows, ensure data quality, and enable discovery and governance across a lakehouse and data mesh.

Work involves Terraform, AWS services, Elastic search of metadata, and collaboration in an agile team to deliver secure, scalable data solutions for government clients.

Qualifications

• 8+ years of professional data engineering experience with production pipelines.
• Hands-on design and operation of streaming and batch data pipelines (Debezium, MSK, Spark, Airflow).
• Strong proficiency in Python and SQL with multi-source data validation.
• Experience with AWS data services and containerized environments delivered through DevSecOps/CI-CD pipelines.

Responsibilities

• Design, build, and operate streaming (Debezium into Amazon MSK) and batch (Spark on AWS EMR) pipelines.
• Validate data quality, lineage, and governance before exposing data to downstream users.
• Maintain metadata/catalogs (AWS Glue) and API Gateway integrations for data discovery.
• Contribute to an agile Kanban/Scrum team, participate in design reviews and code reviews.
• Manage infrastructure as code (Terraform) and secure deployment in line with DoD standards.

Skills

Streaming data pipelines

Debezium

Amazon MSK

Apache Spark

AWS EMR

Python

SQL

Terraform

Kubernetes

Data governance

Tools

Debezium

Amazon MSK

Apache Spark

Apache Airflow

Terraform

AWS Glue

AWS S3

Apache Iceberg

Kubernetes

AWS Aurora

Job description

Security Clearance: Active DoD Secret Clearance

Omni Federal, founded in 2017 and headquartered in Washington, DC, is a highly specialized software solutions provider with a robust presence in key locations across the United States, including Boston, MA, Colorado Springs, CO, San Antonio, TX, and St. Louis, MO. Born out of the Department of Defense's software factory ecosystem, Omni has rapidly distinguished itself by delivering both mission-critical and enterprise solutions that enhance the technological capabilities of the federal government. With a focus on areas such as Command and Control, Cybersecurity, Space, Geospatial, and Modeling & Simulation, Omni leverages cutting-edge commercial technology tailored to government objectives, improving mission performance and delivering transformative outcomes for the Department of Defense (DoD), Intelligence Community (IC), and their end-users. The company's innovative approach is backed by its Omni Labs and SBIR Innovation centers, where they develop advanced platforms and tools in data mesh, secure connectivity, and intelligent automation.

Why Omni?

• Environment of Autonomy
• People over process

We are seeking an experienced Senior Data Engineer to support the Air Force to help in developing and operating modernized applications that are used daily by the warfighter. This is an exciting new initiative where the Air Force is mirroring Silicon Valley processes and capabilities to quickly deploy new capabilities. Candidates must be passionate, energized, and excited to work on modern architectures and solve challenging problems for our clients.

Position Description (what you will do)

• Design, build, and operate streaming (Debezium change-data-capture into Amazon MSK) and batch (Apache Spark via AWS EMR) data pipelines that rapidly onboard new data sources into the program’s Apache Iceberg-based data lakehouse, from initial ingestion through validation and production release.


• Corroborate newly onboarded data against existing data lakehouse holdings to validate quality, consistency, and lineage before it is exposed to downstream consumers.


• Develop and maintain the metadata, cataloging (AWS Glue as the Iceberg data catalog), and API Gateway integrations that make program data easily discoverable and consumable via Amazon Athena, Spark, and other Iceberg-compatible tools by downstream mission partners and vendors through the enterprise data mesh.


• Own assigned technical tasking independently - translate well-articulated requirements into a working solution.


• Engage directly with data providers and data consumers to understand the operational context behind each data source and translate that context accurately for both technical and non-technical audiences.


• Participate in an agile (Kanban/Scrum) delivery team, contributing to sprint planning, technical design discussions, and pipeline/code reviews.


• Maintain pipeline and infrastructure code as Infrastructure-as-Code (Terraform) and follow established DevSecOps build and deployment pipeline processes.


• Support data governance, quality, and auditability practices in accordance with Government-approved data standards for the platform.

Requirements:

• 8+ years of professional data engineering experience, including hands-on design and operation of production data pipelines.


• Demonstrated experience with streaming and batch data pipeline technologies, particularly Debezium (change-data-capture), Amazon MSK (Kafka), Apache Spark (AWS EMR), and Apache Airflow.


• Strong proficiency in Python and SQL, with experience integrating and validating data from multiple heterogeneous sources.


• Experience with AWS data services (e.g., S3, Glue, Aurora) and containerized environments (Kubernetes), delivered through DevSecOps/CI-CD pipelines.


• Demonstrated ability to work independently against well-defined tasking with minimal oversight, and to communicate effectively with both technical and non-technical stakeholders, including data providers and consumers.

Nice to Have:

• Experience supporting DoD or Intelligence Community programs, particularly Platform One or other DoD software factory environments.


• Familiarity with container security and software supply chain concepts (SBOMs, vulnerability scoring and attestation).


• Experience with the Apache Iceberg table format or similar open table formats (e.g., Delta Lake, Hudi) for lakehouse architectures.


• Experience with event-driven architecture (e.g., AWS EventBridge) and Infrastructure-as-Code (Terraform).


• Experience designing or consuming API Gateways in an enterprise data mesh architecture.


• Prior experience in a vendor-or customer-facing data steward or data liaison capacity.

Our team is always hungry for more - more knowledge, more problem solving, more growth, more innovation, and, ultimately, customer success. This hunger and hustle fuel our determination to excel in everything we do. It’s not just about meeting established goals; it’s about exceeding them. We take immense pride in applying our expertise in cutting-edge technologies and our ability to adapt to emerging trends directly to the users in the field, as our nation’s protectors deserve nothing less.

Omni Federal is an equal opportunity and affirmative action employer. Omni Federal is committed to administering all employment and personnel actions on the basis of merit and free of discrimination based on race, color, religion, sex, sexual orientation, gender identity, national origin, protected veteran status, or status as an individual with a disability. Consistent with this commitment, we are dedicated to the employment and advancement of qualified minorities, women, individuals with disabilities, protected veterans, persons of all ethnic backgrounds and religions according to their abilities