Kafka Admin

IMCS Group; St. Louis (MO); On-site; USD 99,000 - 121,000; Full time

Job summary

IMCS Group in Saint Louis, MO is seeking a Kafka Admin to manage enterprise-scale Hadoop, Kafka, and AWS cloud platforms, ensuring high availability, security, and performance across data pipelines.

Responsibilities include deploying and tuning Kafka clusters (including AWS MSK), monitoring lag and health, implementing IaC with Terraform and Ansible, and integrating CI/CD with Jenkins or GitHub Actions. Experience with Kubernetes, Docker, and cloud services is required.

Qualifications

• Experience managing enterprise-scale Kafka deployments and AWS MSK.
• Proficient with security, IAM, and encryption standards.
• Strong automation and IaC skills (Terraform, Ansible, CloudFormation).
• Hands-on with CI/CD, monitoring, and incident response.

Responsibilities

• Deploy, configure, and manage Apache Kafka clusters and AWS MSK environments.
• Monitor broker health, lag, and replication; perform capacity planning.
• Implement security with SSL, SASL, ACLs; ensure compliance.
• Design and maintain CI/CD pipelines; automate deployments with IaC.
• Provide Level 2/3 production support and on-call coverage.

Skills

Kafka Administration

Shell Scripting

Python

Jenkins

Git

Ansible

Terraform

Docker

Kubernetes

Monitoring (Grafana/Prometheus)

Security & HA

Education

Bachelor’s degree in Computer Science or related field

Tools

AWS MSK

Airflow

Spark

Cloudera CDP

Kubernetes

Terraform

Docker

GCP

Kafka

Job description

Visa independent candidate (USC & GC) can work

Max Rate -$55/Hour on w2

Job Title: Kafka Admin

Work Location: Saint Louis, MO 63131

Detailed Job Description

Experienced Big Data Administrator responsible for managing, supporting, automating, and optimizing enterprise-scale Hadoop, Kafka, AWS cloud, and DevOps platforms. The role focuses on ensuring high availability, performance, security, scalability, and operational excellence across big data ecosystems while collaborating with development, architecture, and business teams.

Key Responsibilities:

Kafka Administration

• Deploy, configure, and manage Apache Kafka clusters and AWS MSK environments.
• Monitor broker health, partitions, replication factors, and consumer lag.
• Perform capacity planning and cluster scaling activities.
• Manage Kafka security using SSL, SASL, ACLs, and encryption standards.
• Troubleshoot producer, consumer, and broker performance issues.
• Support Kafka Connect, Schema Registry, Cruise Control, and MirrorMaker implementations.
• Manage cloud infrastructure services including EC2, S3, IAM, VPC, EBS, CloudWatch, CloudTrail and AWS Glue.
• Support AWS Managed Streaming for Kafka (MSK), EMR, Lambda, and Airflow environments.
• Implement cloud security best practices and governance controls.
• Perform infrastructure provisioning and automation using Infrastructure as Code (IaC).
• Monitor cloud resource utilization and optimize operational costs.

DevOps & Automation

• Design and maintain CI/CD pipelines using Jenkins, GitHub Actions, GitLab CI, or similar tools.
• Automate infrastructure deployment using Terraform, CloudFormation, and Ansible.
• Manage source control repositories and release processes.
• Implement monitoring and alerting solutions using Prometheus, Grafana, Splunk, ELK, or CloudWatch.
• Support containerization technologies such as Docker and Kubernetes.
• Develop automation scripts using Python, Shell, or Bash.

Operations & Support

• Provide Level 2 and Level 3 production support.
• Participate in on-call support rotations and incident management activities.
• Perform root cause analysis (RCA) and implement preventive measures.
• Create and maintain operational documentation and standard operating procedures.
• Ensure compliance with security, audit, and regulatory requirements.

Required Skills

• KRAFT and MSK
• Shell Scripting and Python
• Jenkins, Git, Ansible, Terraform
• Docker and Kubernetes
• Monitoring Tools (Grafana, Prometheus, Splunk) Networking, Security, and High Availability Concepts Performance Tuning and Capacity Planning

Preferred Qualifications:

• Bachelor’s degree in computer science, Information Technology, or related field.
• Experience with Cloudera CDP, AWS MSK, Airflow, and Spark.
• AWS, GCP, Kafka, or Kubernetes certifications.
• Experience supporting large-scale production environments handling petabyte-scale data workloads.

Key Achievements Expected

• Maintain platform availability above 99.9%.
• Adopt AI-assisted engineering practices to improve operational efficiency, reduce manual effort, and accelerate troubleshooting and documentation.
• Improve cluster performance and resource utilization.
• Ensure secure, scalable, and reliable data platform operations.
• Support enterprise data engineering, analytics, and AI/ML workloads efficiently.