Data Engineer (ELK)

Red River Computer Co; O'Fallon (IL); On-site; USD 120,000 - 160,000; Full time

Job summary

Red River Computer Co. seeks a data engineer software developer to design, build, and maintain scalable data pipelines for analytics at Scott Air Force Base.

You will ingest, transform, and unify data from DoD networks while ensuring data quality and compliance with DISA requirements. Responsibilities include implementing Kafka/Elastic Stack solutions, developing data warehouses/lakes, and optimizing performance and cost.

Qualifications

• Bachelor’s degree in CS/IT/Engineering or related field.
• ≥5 years of data engineering/data warehousing experience.
• Active DoD Secret Clearance and Security+ certification required.
• Experience with Kafka, Spark, NiFi, Elasticsearch, Logstash, Databricks, and ELK Stack.
• Cloud platforms (AWS, Azure, GCP; GovCloud) and IaC with networking/security policies.
• Kubernetes, Ansible/Puppet/Chef; monitoring with Prometheus/Grafana/Elastic.
• BI visualization with Kibana; strong documentation and collaboration skills.
• Agile methodologies (Scrum/SAFe); Jira and Confluence familiarity.

Responsibilities

• Designs, builds, and maintains scalable data systems and pipelines for analytics and reporting.
• Unifies data from multiple network operations systems into a consistent, accessible platform with data quality and security.
• Implements solutions using Confluent Kafka, Elastic Stack, AWS, and OpenShift; creates Kibana dashboards with React/JS/HTML.
• Develops data warehouses and data lakes; optimizes performance and cost through automation.
• Optimizes ingestion throughput, query performance, indexing, and storage across Elastic and Kafka.
• Automates ETL/ELT, data validation, monitoring; supports CI/CD pipelines.
• Collaborates with data analysts and visualization engineers; documents architecture and procedures.
• Troubleshoots data ingestion, processing, and platform performance issues.

Skills

Data pipeline design

Distributed systems

Security clearance

Communication

Agile methodologies

Education

Bachelor's degree in computer science, information technology, engineering, or related field

Security+ certification

Active DoD Secret Clearance

Tools

Kafka

Spark

NiFi

Hadoop

Elasticsearch

Logstash

Databricks

ELK Stack

Kubernetes

AWS

Azure

GCP

OpenShift

Kibana

React

JavaScript/HTML

Jira

Confluence

Confluent Developer Certification

Elastic Engineer Certification

Job description

We are seeking a highly skilled and motivated Data Engineer Software Developer to join our team at Scott Air Force Base, supporting the Defense Information Systems Agency (DISA). The successful candidate will play a crucial role in designing, developing, and implementing solutions. As well as maintaining visualizations to support data analytics and reporting requirements. This position requires a deep understanding of data visualization principles, strong analytical skills, and the ability to work collaboratively in a fast-paced environment.

Primary Responsibilities

• Designs, builds, and maintains scalable, reliable data systems and pipelines to ingest, process, and transform large, complex, and disparate data sources for analytics and reporting.
• Unifies and integrates data from multiple network operations systems into a consistent, accessible platform, ensuring data quality, accuracy, security, and compliance with DoD and DISA requirements.
• Implement solutions using Confluent Kafka, Elastic Stack, AWS, and Red Hat OpenShift (Kubernetes), including creating Kafka, Elastic, and Logstash pipelines and supporting Kibana visualizations and dashboards with React, JavaScript, and HTML.
• Develops and manage data warehouses and data lakes to support analytics, reporting, and operational dashboards, optimizing infrastructure for performance, scalability, and cost-effectiveness through automation and best practices.
• Continually analyze and optimize ingestion throughput, query performance, indexing strategies, and storage efficiency across Elastic and Kafka platforms.
• Automates data-related tasks such as ETL/ELT processes, data validation, cleansing, monitoring, and supports CI/CD pipelines for automated build, test, and deployment.
• Collaborates with data analysts, visualization engineers, and other stakeholders to deliver integrated, actionable data solutions, and documents data architecture, integration patterns, and operational procedures.
• Troubleshoots and resolves issues related to data ingestion, processing, and platform performance, storage, memory, partitioning, and cluster performance.

Qualifications

• Bachelor’s degree in computer science, Information Technology, Engineering, or a related field or equivalent experience.
• Minimum of 5 years of experience in data engineering, data warehousing, or a related field.
• Active DoD Secret Clearance and current Security+ certification is required.
• Experience with DISA, DISN, or other DoD network environments, including developing and deploying software applications that meet DoD security standards (e.g., STIGs), and knowledge of network security, encryption, and access controls in classified environments.
• Experience with big data technologies and platforms such as Kafka, Spark, NiFi, Hadoop, Elasticsearch, Logstash, Databricks, and ELK Stack for text mining, summarization, search, and entity extraction.
• Experience with cloud-based data platforms (AWS, Azure, GCP, AWS GovCloud), cloud-integrated platforms, and infrastructure as code, including networking and security policies.
• Familiarity with Kubernetes deployment, platform upgrades, patching, configuration management tools (Ansible, Puppet, Chef), and monitoring, alerting, and observability tools (Prometheus, Grafana, Elastic).
• Experience with BI tools like Kibana and developing and maintaining technical documentation and runbooks.
• Strong understanding and practical experience with Agile methodologies (Scrum, SAFe), and related tools such as Atlassian JIRA and Confluence.
• Certified Confluent Developer and Certified Elastic Engineer.
• Excellent communication and collaboration skills, with experience working remotely with geographically dispersed teams and within a matrixed organization to combine software, integration, and data engineering practices.
• Apply knowledge of system architecture, networks, and centralized logging (ELK) to support data transformation initiatives.

Basic Qualifications

• U.S. Citizenship Required

Red River offers a competitive salary, excellent benefits and an exceptional work environment. You can review our benefit offerings here.

Red River does not accept unsolicited resumes from individual recruiters or third-party recruiting agencies in response to job postings or otherwise. Placement fees will not be paid to any recruiter unless Red River has an active agreement in place with the recruiter and such a request has been made by the Red River Talent Acquisition team and such candidate was submitted to the Red River Talent Acquisition Team via our Applicant Tracking System. Any unsolicited resumes or other data submitted to Red River in violation of this policy may be used by Red River without obligation to pay any fees of any kind to the recruiter.

EOE M/F/DISABLED/Vet Red River is an equal opportunity employer. All qualified applicants will receive consideration for employment. Discrimination or harassment based upon any protected characteristics as defined by state or federal law is wholly inconsistent with our company values and will not be tolerated.

In order to ensure reasonable accommodation for individuals protected by Section 503 of the Rehabilitation Act of 1973, the Vietnam Veterans Readjustment Act of 1974, and Title I of the American’s with Disabilities Act of 1990, applicants that require accommodation in the job application process may contact [e-mail].

https://redriver.com/wp-content/uploads/2023/08/CPRA-Notice-to-Employees-updated30.pdf https://www.dol.gov/agencies/ofccp/posters