Senior Data Engineer - ELK, Kafka & Cloud Pipelines

Red River; O'Fallon (IL); On-site; USD 120,000 - 180,000; Full time

Job summary

Red River is seeking a highly skilled Data Engineer to design, build, and maintain scalable data pipelines at Scott Air Force Base, supporting the Defense Information Systems Agency (DISA). The role emphasizes ingestion, processing, and analytics for reporting while upholding DoD security standards.

You will work with Kafka, Elastic Stack, AWS, and Kubernetes, delivering Kibana dashboards and data-driven insights.

Qualifications

• Bachelor’s degree or equivalent experience in a related field.
• Minimum 5 years in data engineering, data warehousing, or related.
• Active DoD Secret Clearance and Security+ certification required.
• Experience with DoD networks and security standards (STIGs).
• Experience with Kafka, Spark, NiFi, Hadoop, Elasticsearch, Logstash, Databricks, ELK Stack.

Responsibilities

• Design, build, and maintain scalable data systems and pipelines for analytics and reporting.
• Unify data from multiple networks into a consistent platform with data quality and security.
• Implement solutions using Kafka, Elastic Stack, AWS, and Kubernetes, with Kibana visualizations.
• Develop data warehouses/lakes to support analytics and dashboards.
• Optimize ingestion throughput, query performance, and storage efficiency.
• Automate ETL/ELT processes and support CI/CD pipelines.

Skills

Big data pipelines

Kubernetes

Kafka

Elasticsearch

Pipelines optimization

CI/CD

Communication

Education

Bachelor’s degree in CS/IT/Engineering

Tools

Kafka

Spark

NiFi

Hadoop

Elasticsearch/ELK

Logstash

Databricks

Kubernetes

Job description

Red River is seeking a highly skilled Data Engineer to design, build, and maintain scalable data pipelines at Scott Air Force Base, supporting the Defense Information Systems Agency (DISA). The role emphasizes ingestion, processing, and analytics for reporting while upholding DoD security standards.

You will work with Kafka, Elastic Stack, AWS, and Kubernetes, delivering Kibana dashboards and data-driven insights.