Mission-Critical Data Engineer: Kafka/NiFi & ETL

Insight Global; Hanahan (SC); Hybrid; USD 100,000 - 150,000; Full time

Job summary

Insight Global is seeking a fast-learning Data Engineer with Apache Kafka and NiFi experience for a hybrid role in North Charleston, SC. You will design, build, and maintain data systems for a federal program, supporting ETL pipelines, infrastructure automation, and data architectures.

The role emphasizes collaboration across engineering teams, DevOps practices, and strong documentation to align data infrastructure with mission objectives.

Qualifications

• 3–5 years of experience supporting data engineering efforts.
• 5+ years of experience in data engineering roles.
• Experience with containerization and orchestration tools (Docker, Kubernetes).
• Experience with CI/CD tools (Jenkins, GitLab CI, Azure DevOps, GitHub Actions).
• Experience with distributed event streaming platforms (Apache Kafka, Debezium).
• Experience with Apache NiFi for data orchestration.
• Experience with data storage and caching services (Postgres, Redis, MinIO).
• Proficiency with Python, PowerShell, Bash, or Go.

Responsibilities

• Design, build, and maintain ETL pipelines to move data from disparate sources into centralized repositories.
• Develop automated deployment pipelines and IaC templates to automate data tasks.
• Design and optimize data architectures for performance, cost-efficiency, security, and scalability.
• Design and implement large-scale processing systems and data lake/warehouse solutions.
• Prepare technical documentation and present architecture proposals to engineering teams.
• Collaborate with other engineering groups to align infrastructure with mission objectives.
• Provide recommendations for data engineering best practices and process improvements.

Skills

Data engineering

Docker

Kubernetes

CI/CD

Python

PowerShell/Bash/Go

Apache Kafka

Apache NiFi

Grafana/ELK

Nginx/Apisix

Education

Bachelor's degree in Computer Science or related field

Tools

Docker

Kubernetes

Helm

Jenkins/GitHub Actions/Azure DevOps

Git

Ansible/Chef/Puppet

Postgres/Redis/MinIO

Job description

Insight Global is seeking a fast-learning Data Engineer with Apache Kafka and NiFi experience for a hybrid role in North Charleston, SC. You will design, build, and maintain data systems for a federal program, supporting ETL pipelines, infrastructure automation, and data architectures.

The role emphasizes collaboration across engineering teams, DevOps practices, and strong documentation to align data infrastructure with mission objectives.