Real-Time Data Engineer – Kafka & Elastic Pipelines

Leidos; Shiloh (IL); On-site; USD 120,000 - 180,000; Full time

Job summary

Leidos is seeking a Data Engineer Software Developer to join the RTV team on the GAINS program. You will design, build, and maintain scalable data systems to ingest, store, and transform large data volumes from DoD networks.

You will integrate sources with Confluent Kafka and Elastic Stack, develop data warehouses, and optimize performance while following CI/CD and governance practices. Onsite at Scott AFB, with clearance requirements.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related technical field plus 4+ years of relevant experience.
• Hands-on experience designing and maintaining data pipelines (ETL/ELT) across disparate sources.
• Experience with data warehouses/lakes (AWS Redshift, S3, Hadoop, Snowflake).
• Proficiency in Python, Java, SQL, and shell scripting.
• Experience with Elastic Stack and/or Kafka for data ingestion and processing.
• Familiarity with DoD/ DISA data security and RMF/ STIGs.

Responsibilities

• Designs, builds, and maintains scalable data systems and pipelines for analytics.
• Unifies data from multiple network operations sources into a consistent platform.
• Implements pipelines using Kafka, Elastic Stack, AWS, and OpenShift; builds Kibana dashboards.
• Develops data warehouses/data lakes to support analytics and dashboards.
• Optimizes ingestion throughput and query performance on Elastic and Kafka.
• Automates ETL/ELT tasks and supports CI/CD pipelines.
• Collaborates with analysts and developers; documents architecture and procedures.
• Troubleshoots data ingestion, processing, and platform performance issues.

Skills

Data pipelines

ETL/ELT

Kafka

Elastic Stack

Python

SQL

AWS

CI/CD

Linux

DoD compliance

Education

Bachelor’s degree in Computer Science or related

Tools

Kafka

Elastic Stack

AWS

OpenShift

Job description

Leidos is seeking a Data Engineer Software Developer to join the RTV team on the GAINS program. You will design, build, and maintain scalable data systems to ingest, store, and transform large data volumes from DoD networks.

You will integrate sources with Confluent Kafka and Elastic Stack, develop data warehouses, and optimize performance while following CI/CD and governance practices. Onsite at Scott AFB, with clearance requirements.