Top Secret Data Engineer - Apache

Insight Global; Hanahan (SC); Hybrid; USD 100,000 - 150,000; Full time

Job summary

Insight Global is seeking a fast-learning Data Engineer with Apache Kafka and NiFi experience for a hybrid role in North Charleston, SC. You will design, build, and maintain data systems for a federal program, supporting ETL pipelines, infrastructure automation, and data architectures.

The role emphasizes collaboration across engineering teams, DevOps practices, and strong documentation to align data infrastructure with mission objectives.

Qualifications

• 3–5 years of experience supporting data engineering efforts.
• 5+ years of experience in data engineering roles.
• Experience with containerization and orchestration tools (Docker, Kubernetes).
• Experience with CI/CD tools (Jenkins, GitLab CI, Azure DevOps, GitHub Actions).
• Experience with distributed event streaming platforms (Apache Kafka, Debezium).
• Experience with Apache NiFi for data orchestration.
• Experience with data storage and caching services (Postgres, Redis, MinIO).
• Proficiency with Python, PowerShell, Bash, or Go.

Responsibilities

• Design, build, and maintain ETL pipelines to move data from disparate sources into centralized repositories.
• Develop automated deployment pipelines and IaC templates to automate data tasks.
• Design and optimize data architectures for performance, cost-efficiency, security, and scalability.
• Design and implement large-scale processing systems and data lake/warehouse solutions.
• Prepare technical documentation and present architecture proposals to engineering teams.
• Collaborate with other engineering groups to align infrastructure with mission objectives.
• Provide recommendations for data engineering best practices and process improvements.

Skills

Data engineering

Docker

Kubernetes

CI/CD

Python

PowerShell/Bash/Go

Apache Kafka

Apache NiFi

Grafana/ELK

Nginx/Apisix

Education

Bachelor's degree in Computer Science or related field

Tools

Docker

Kubernetes

Helm

Jenkins/GitHub Actions/Azure DevOps

Git

Ansible/Chef/Puppet

Postgres/Redis/MinIO

Job description

Job Description

A rapidly expanding company supporting federal and military programs is seeking a fast-learning and self-motivated Data Engineer with Apache Kafka/NiFi experience to design, build, and maintain systems and infrastructure that collect, store, and transform data for a mission‑critical federal program. This effort is primarily centered around Tactical Network data engineering support. This is a hybrid role in North Charleston, SC.

You Will:

• Support designing and building ETL (Extract, Transform, Load) processes and pipelines to move data from disparate sources into centralized repositories.
• Develop automated deployment pipelines and Infrastructure-as-Code templates using scripts and orchestration tools to automate repetitive data tasks, reducing manual intervention and error.
• Design and optimize data architectures to improve performance, cost‑efficiency, security, and scalability.
• Design and implement large‑scale processing systems and database architecture solutions (e.g., Data Lakes, Data Warehouses).
• Provide recommendations for data engineering best practices and process improvements.
• Contribute to team objectives and support data engineering initiatives.
• Prepare technical documentation and present architecture proposals to engineering teams.
• Partner with other engineering groups to understand their data requirements and with leadership to align technical infrastructure with mission objectives.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances.

If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to [e-mail].

To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

Skills and Requirements

• Possession of and the ability to maintain a Secret Security Clearance.
• 3-5 years of experience supporting data engineering efforts and initiatives.
• Experience with containerization technologies (Docker, Kubernetes) is highly valued.
• Experience with Helm charts and deploying, managing, and scaling applications.
• Working knowledge of Agile methodology and DevOps practices.
• Experience with open-source, distributed event streaming platforms used to collect, process, store, and analyze data feeds (Apache Kafka).
• Experience with data orchestration tools (Apache NiFi).
• Experience with open-source monitoring and observability architectures including Grafana LGTM+ Stack and Elastic (ELK) Stack.
• Experience with high-performance, open-source networking infrastructure tools (Nginx, Apache APISIX).
• Experience with open-source, distributed event streaming platforms used to collect, process, store, and analyze data feeds (Apache Kafka, Debezium).
• Experience with data orchestration tools (Apache NiFi).
• Experience with open-source data storage and caching services (Postgres, Redis, MinIO).
• Strong documentation and communication skills to work effectively across technical teams.
• Bachelor's degree preferred or equivalent experience.
• Bachelor's degree in Computer Science, Engineering, or related field.
• 5+ years of experience in data engineering roles.
• Experience with CI/CD tools (Jenkins, GitLab CI, Azure DevOps, GitHub Actions).
• Proficiency with version control systems and collaborative development tools
• Experience with configuration management and automation platforms (Ansible, Chef, Puppet).
• Programming/scripting experience (Python, PowerShell, Bash, Go).
• Experience with data engineering design and visualization tools (Swagger).
• Knowledge of microservices architecture and serverless computing