Real-Time Data Engineer (Kafka/Elastic)

Leidos Inc; Illinois; On-site; USD 87,000 - 157,000; Full time

Job summary

Leidos is seeking a Data Engineer Software Developer to join the RTV team within GAINS, focusing on data ingestion, processing, and analytics across the Elastic/Kafka stack. Role requires building scalable data pipelines and integrating diverse data sources for a unified UX/UI on the Elastic platform.

Optimal candidates will have DoD/ DISA familiarity, strong Python/Java/SQL, CI/CD, and cloud experience. Onsite work near Scott AFB, IL with Secret clearance requirements.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or related technical discipline with 4+ years of relevant experience.
• Hands-on experience designing, building, and maintaining data pipelines (ETL/ELT).
• Experience with data warehouse and/or data lake technologies (e.g., AWS Redshift, S3, Hadoop, Snowflake).
• Proficiency with Python, Java, SQL, and shell scripting.
• Experience with Elastic Stack and/or Kafka for data ingestion and processing.
• Familiarity with network operations data (logs, metrics, events, ticketing, CMDB) and data models.
• Experience with Linux/UNIX system administration and automation.
• Knowledge of data security, access controls, and compliance (DoD, DISA, RMF, STIGs).
• Experience with CI/CD pipelines, containerized pipelines, Git, and DevOps practices.
• Strong communication and documentation skills.

Responsibilities

• Designs, builds, and maintains scalable data systems and pipelines to ingest, process, and transform large, complex data sources.
• Unifies and integrates data from multiple network operations systems into a consistent platform with data quality and security.
• Implements solutions using Confluent Kafka, Elastic Stack, AWS, and OpenShift/Kubernetes, including pipelines and Kibana visualizations.
• Develops and manages data warehouses and data lakes for analytics and dashboards, optimizing for performance and cost.
• Continually analyze and optimize ingestion throughput, query performance, indexing strategies, and storage across Elastic and Kafka.
• Automates ETL/ELT processes, data validation, cleansing, monitoring, and supports CI/CD pipelines.
• Collaborates with data analysts and visualization engineers to deliver integrated data solutions and document architectures.
• Troubleshoots data ingestion, processing, and platform performance issues.

Skills

ETL pipelines

Data integration

Python

Java

SQL

Shell scripting

Elastic Stack

Kafka

AWS

Kubernetes

Linux/UNIX

Security compliance

CI/CD

DevOps

Data warehouse

DoD/DISA knowledge

Git/Bitbucket

Documentation

Communication

Education

Bachelor's degree in Computer Science/Engineering

Tools

Elastic Stack

Kafka

OpenShift

Kubernetes

Databricks

NiFi

Snowflake

Redshift

Hadoop

ELK Stack

Job description

Leidos is seeking a Data Engineer Software Developer to join the RTV team within GAINS, focusing on data ingestion, processing, and analytics across the Elastic/Kafka stack. Role requires building scalable data pipelines and integrating diverse data sources for a unified UX/UI on the Elastic platform.

Optimal candidates will have DoD/ DISA familiarity, strong Python/Java/SQL, CI/CD, and cloud experience. Onsite work near Scott AFB, IL with Secret clearance requirements.