Data Engineer — ELK Stack & Real-Time Analytics

Red River Computer Co; O'Fallon (IL); On-site; USD 120,000 - 160,000; Full time

Job summary

Red River Computer Co. seeks a data engineer software developer to design, build, and maintain scalable data pipelines for analytics at Scott Air Force Base.

You will ingest, transform, and unify data from DoD networks while ensuring data quality and compliance with DISA requirements. Responsibilities include implementing Kafka/Elastic Stack solutions, developing data warehouses/lakes, and optimizing performance and cost.

Qualifications

• Bachelor’s degree in CS/IT/Engineering or related field.
• ≥5 years of data engineering/data warehousing experience.
• Active DoD Secret Clearance and Security+ certification required.
• Experience with Kafka, Spark, NiFi, Elasticsearch, Logstash, Databricks, and ELK Stack.
• Cloud platforms (AWS, Azure, GCP; GovCloud) and IaC with networking/security policies.
• Kubernetes, Ansible/Puppet/Chef; monitoring with Prometheus/Grafana/Elastic.
• BI visualization with Kibana; strong documentation and collaboration skills.
• Agile methodologies (Scrum/SAFe); Jira and Confluence familiarity.

Responsibilities

• Designs, builds, and maintains scalable data systems and pipelines for analytics and reporting.
• Unifies data from multiple network operations systems into a consistent, accessible platform with data quality and security.
• Implements solutions using Confluent Kafka, Elastic Stack, AWS, and OpenShift; creates Kibana dashboards with React/JS/HTML.
• Develops data warehouses and data lakes; optimizes performance and cost through automation.
• Optimizes ingestion throughput, query performance, indexing, and storage across Elastic and Kafka.
• Automates ETL/ELT, data validation, monitoring; supports CI/CD pipelines.
• Collaborates with data analysts and visualization engineers; documents architecture and procedures.
• Troubleshoots data ingestion, processing, and platform performance issues.

Skills

Data pipeline design

Distributed systems

Security clearance

Communication

Agile methodologies

Education

Bachelor's degree in computer science, information technology, engineering, or related field

Security+ certification

Active DoD Secret Clearance

Tools

Kafka

Spark

NiFi

Hadoop

Elasticsearch

Logstash

Databricks

ELK Stack

Kubernetes

AWS

Azure

GCP

OpenShift

Kibana

React

JavaScript/HTML

Jira

Confluence

Confluent Developer Certification

Elastic Engineer Certification

Job description

Red River Computer Co. seeks a data engineer software developer to design, build, and maintain scalable data pipelines for analytics at Scott Air Force Base.

You will ingest, transform, and unify data from DoD networks while ensuring data quality and compliance with DISA requirements. Responsibilities include implementing Kafka/Elastic Stack solutions, developing data warehouses/lakes, and optimizing performance and cost.