Data Engineer – ELK, Kafka & Data Pipelines

redriver; O'Fallon (IL); On-site; USD 120,000 - 180,000; Full time

Benefits offered by this job

Competitive benefits

Job summary

Red River seeks a Data Engineer/Software Developer at Scott Air Force Base to design, build, and maintain scalable data pipelines for analytics and reporting. You will integrate data from multiple DoD systems, ensure data quality and security, and implement solutions using Kafka, Elastic Stack, AWS, and OpenShift with Kibana dashboards.

The role requires 5+ years in data engineering, DoD experience, and active clearance.

Qualifications

• Bachelor's degree in computer science, information technology, engineering, or a related field.
• Minimum of 5 years of experience in data engineering, data warehousing, or a related field.
• Active DoD Secret Clearance and Security+ certification is required.
• Experience with DISA/DoD environments and security standards (STIGs).
• Experience with big data technologies: Kafka, Spark, NiFi, Hadoop, Elasticsearch, Logstash, Databricks, and ELK Stack.
• Experience with cloud platforms (AWS, Azure, GCP, AWS GovCloud) and infrastructure as code.
• Familiarity with Kubernetes deployment, Ansible/Puppet/Chef, and monitoring tools.
• Experience with Kibana and data visualization.
• Strong understanding of Agile methodologies and tools (JIRA, Confluence).
• Certified Confluent Developer and Certified Elastic Engineer.

Responsibilities

• Designs, builds, and maintains scalable data systems and pipelines to ingest, process, and transform data for analytics and reporting.
• Unifies data from multiple network operations systems into a consistent platform with data quality, security, and compliance.
• Implement solutions using Confluent Kafka, Elastic Stack, AWS, and OpenShift (Kubernetes) with Kibana visualizations and dashboards (React/JS/HTML).
• Develops and manage data warehouses and data lakes to support analytics and dashboards, optimizing performance and cost.
• Analyze and optimize ingestion throughput, query performance, and storage across Elastic and Kafka platforms.
• Automates ETL/ELT processes and supports CI/CD pipelines for automated build, test, deployment.
• Collaborates with analysts and visualization engineers to deliver integrated data solutions and documents architecture and procedures.
• Troubleshoot and resolve data ingestion, processing, and platform performance issues.

Skills

Data integration

Data pipelines

Visualization

Agile methodologies

Collaborative work

Security & compliance

Cloud platforms

CI/CD

React

JavaScript

Education

Bachelor's degree in Computer Science / IT / Engineering

Tools

Kafka

Elasticsearch

Logstash

Kibana

React

JavaScript

HTML

AWS

OpenShift

Databricks

Spark

NiFi

Hadoop

ELK Stack

Job description

Red River seeks a Data Engineer/Software Developer at Scott Air Force Base to design, build, and maintain scalable data pipelines for analytics and reporting. You will integrate data from multiple DoD systems, ensure data quality and security, and implement solutions using Kafka, Elastic Stack, AWS, and OpenShift with Kibana dashboards.

The role requires 5+ years in data engineering, DoD experience, and active clearance.