Data Engineer

Insight Global; Charleston (SC); Hybrid; USD 100,000 - 140,000; Full time

Job summary

Insight Global in Charleston, SC, is seeking a Data Engineer with Apache Kafka and NiFi experience to design, build, and maintain data systems for a mission-critical federal program. This hybrid role focuses on Tactical Network data engineering support.

The candidate will design ETL pipelines, deploy IaC templates, optimize architectures for performance and security, and collaborate with multiple engineering teams to align infrastructure with program objectives. A Secret clearance is required.

Qualifications

• Secret Security Clearance required.
• 3–5 years of data engineering experience.
• Experience with Kafka and NiFi.
• Experience with containerization (Docker, Kubernetes).
• Familiar with Helm charts and deploying applications.
• Proficient in Agile/DevOps practices.
• Experience with data storage (Postgres, Redis, MinIO).
• Strong documentation and cross-team collaboration.

Responsibilities

• Design and build ETL processes and data pipelines.
• Develop automated deployment pipelines and IaC templates.
• Design and optimize data architectures for performance and security.
• Create large-scale processing systems and data warehouses/l Lakes.
• Provide data engineering best-practices recommendations.
• Support data engineering initiatives and team objectives.
• Prepare technical documentation and present architectures.
• Collaborate with other engineering groups to align data needs with objectives.

Skills

Data engineering

Kafka

NiFi

Docker

Kubernetes

Helm charts

CI/CD

Agile

DevOps

Security clearance

Postgres

Redis

MinIO

Nginx

ELK

Swagger

Python/Go

Education

Bachelor's degree in Computer Science or related field

Bachelor's degree or equivalent work experience

Tools

Docker

Kubernetes

Helm charts

PostgreSQL

Redis

MinIO

Nginx

APISIX

Grafana

ELK

Swagger

Python

Go

Git

CI/CD tools

Job description

A rapidly expanding company supporting federal and military programs is seeking a fast-learning and self-motivated Data Engineer with Apache Kafka/NiFi experience to design, build, and maintain systems and infrastructure that collect, store, and transform data for a mission-critical federal program. This effort is primarily centered around Tactical Network data engineering support. This is a hybrid role in North Charleston, SC.

You Will:

• - Support designing and building ETL (Extract, Transform, Load) processes and pipelines to move data from disparate sources into centralized repositories.
• - Develop automated deployment pipelines and Infrastructure-as-Code templates using scripts and orchestration tools to automate repetitive data tasks, reducing manual intervention and error.
• - Design and optimize data architectures to improve performance, cost-efficiency, security, and scalability.
• - Design and implement large-scale processing systems and database architecture solutions (e.g., Data Lakes, Data Warehouses).
• - Provide recommendations for data engineering best practices and process improvements.
• - Contribute to team objectives and support data engineering initiatives.
• - Prepare technical documentation and present architecture proposals to engineering teams.
• - Partner with other engineering groups to understand their data requirements and with leadership to align technical infrastructure with mission objectives.

Required Skills & Experience

• - Possession of and the ability to maintain a Secret Security Clearance.
• - 3–5 years of experience supporting data engineering efforts and initiatives.
• - Experience with containerization technologies (Docker, Kubernetes) is highly valued.
• - Experience with Helm charts and deploying, managing, and scaling applications.
• - Working knowledge of Agile methodology and DevOps practices.
• - Experience with open-source, distributed event streaming platforms used to collect, process, store, and analyze data feeds (Apache Kafka).
• - Experience with data orchestration tools (Apache NiFi).
• - Experience with open-source monitoring and observability architectures including Grafana LGTM+ Stack and Elastic (ELK) Stack.
• - Experience with high-performance, open-source networking infrastructure tools (Nginx, Apache APISIX).
• - Experience with open-source, distributed event streaming platforms used to collect, process, store, and analyze data feeds (Apache Kafka, Debezium).
• - Experience with data orchestration tools (Apache NiFi).
• - Experience with open-source data storage and caching services (Postgres, Redis, MinIO).
• - Strong documentation and communication skills to work effectively across technical teams.
• - Bachelor's degree preferred or equivalent experience.

Nice to Have Skills & Experience

• - Bachelor's degree in Computer Science, Engineering, or related field.
• - 5+ years of experience in data engineering roles.
• - Experience with CI/CD tools (Jenkins, GitLab CI, Azure DevOps, GitHub Actions).
• - Proficiency with version control systems and collaborative development tools
• - Experience with configuration management and automation platforms (Ansible, Chef, Puppet).
• - Programming/scripting experience (Python, PowerShell, Bash, Go).
• - Experience with data engineering design and visualization tools (Swagger).
• - Knowledge of microservices architecture and serverless computing