Kafka Administrator

Droisys; St. Louis (MO); On-site; USD 171,924,000 - 186,252,000; Full time

Job summary

Droisys is migrating a portfolio of Apache Kafka clusters from on-premises platforms (Cloudera and Confluent) to AWS MSK, managed via Terraform and GitLab CI/CD. We seek a senior, hands-on Kafka engineer to own day-to-day administration across environments, drive the migration, and coordinate with producer/consumer teams to ensure a smooth cutover.

This is a senior IC role for someone who has run Kafka at scale in production, in a hybrid on-prem/cloud setup, and who can be the technical contact

Qualifications

• 5+ years in data/platform/infrastructure engineering.
• 4+ years operating Apache Kafka in production.
• Hands-on administration with Cloudera and Confluent Platform.
• Production experience with AWS MSK (IAM auth, networking).
• Experience migrating Kafka workloads to the cloud with minimal disruption.
• Terraform and GitLab CI/CD for automation.
• Datadog for monitoring, dashboards, and alerting.
• Deep understanding of Kafka internals and delivery semantics.
• Solid AWS fundamentals and Linux scripting.
• Strong communication across producer/consumer teams.

Responsibilities

• Kafka Administration (On‑Prem + Cloud): manage clusters, topics, partitions, replication, quotas, ACLs, and consumer groups.
• Tune brokers, producers, and consumers for throughput and reliability.
• Handle patching, upgrades, capacity planning, and health checks.
• Configure security: TLS/SSL, SASL, IAM, RBAC, encryption.
• Migration to AWS MSK: cutover planning, minimal downtime, data integrity checks.
• CI/CD & IaC: Terraform and GitLab pipelines for deployments.
• Observability: Datadog dashboards, alerts, and SLA tracking.
• Producer/Consumer coordination: liaison between platform and applications.
• Guide client configuration, partitioning, and exactly-once semantics.
• Coordinate migration schedules and testing with teams.

Skills

Kafka administration

Terraform

GitLab CI/CD

Datadog monitoring

AWS MSK

Linux scripting

Networking basics

Communication

Production Kafka

Cluster tuning

Tools

Cloudera

Confluent Platform

AWS MSK

Kubernetes

Job description

Droisys is an innovation technology company focused on helping companies accelerate their digital initiatives from strategy and planning through execution. We leverage deep technical expertise, Agile methodologies, and data-driven intelligence to modernize systems of engagement and simplify human/tech interaction.

Amazing things happen when we work in environments where everyone feels a true sense of belonging and when candidates have the requisite skills and opportunities to succeed. At Droisys, we invest in our talent and support career growth, and we are always on the lookout for amazing talent who can contribute to our growth by delivering top results for our clients. Join us to challenge yourself and accomplish work that matters.

Here are the job details

Job Title Kafka Admin

Duration 12 Months

Rate: $60/hr - $65/hr C2C

About the Role

We are migrating a portfolio of Apache Kafka clusters from on-premises platforms (Cloudera and Confluent) to AWS MSK, managed through a CI/CD pipeline built on Terraform and GitLab. We are looking for a senior, hands‑on Kafka engineer to own day‑to‑day Kafka administration across both environments, drive the migration, and coordinate closely with producer and consumer application teams to ensure a smooth, low‑risk cutover.

This is a senior individual‑contributor role for someone who has run Kafka at scale in production, is comfortable in a hybrid on‑prem/cloud world, and can act as the technical point of contact between platform and application teams.

Key Responsibilities

Kafka Administration (On‑Prem + Cloud)

Administer and operate Kafka clusters across Cloudera, Confluent Platform, and AWS MSK.

Manage topics, partitions, replication, retention, quotas, ACLs, and consumer groups.

Tune brokers, producers, and consumers for throughput, latency, and reliability.

Handle patching, upgrades, capacity planning, and cluster health/performance troubleshooting.

Configure and maintain security: TLS/SSL, SASL, mTLS, IAM (for MSK), RBAC/ACLs, and encryption at rest and in transit.

Migration to AWS MSK

Plan and execute migration of clusters and workloads from on‑prem (Cloudera/Confluent) to AWS MSK.

Design cutover strategies (e.g., MirrorMaker 2 / Confluent Replicator / cluster linking) with minimal downtime and no data loss.

Validate topic parity, offsets, throughput, and data integrity pre- and post-migration.

Define rollback plans and run migration dry‑runs in lower environments.

CI/CD & Infrastructure as Code

Build and maintain Kafka/MSK infrastructure using Terraform.

Manage automated deployments and configuration through GitLab CI/CD pipelines.

Implement GitOps practices for topic, ACL, and cluster configuration management.

Observability & Monitoring

Build and maintain monitoring, alerting, and dashboards in Datadog for Kafka/MSK (broker metrics, consumer lag, partition health, JVM/GC, throughput).

Establish SLAs/SLOs and proactive alerting to reduce incidents and mean‑time‑to‑resolution.

Correlate Kafka metrics with application and infrastructure telemetry.

Producer / Consumer Coordination

Serve as the primary technical liaison between the Kafka platform team and producer/consumer application teams.

Advise application teams on client configuration, serialization/schema management, partitioning strategy, idempotence, and exactly‑once/at‑least‑once semantics.

Support onboarding of new producers/consumers and troubleshoot client‑side issues (rebalancing, lag, poison messages).

Coordinate migration schedules, testing, and cutover windows with each impacted team.

Required Qualifications

5+ years in data/platform/infrastructure engineering, with 4+ years operating Apache Kafka in production.

Hands‑on administration experience with Cloudera and Confluent Platform.

Production experience with AWS MSK (provisioning, configuration, IAM auth, networking).

Proven experience migrating Kafka workloads to the cloud with minimal disruption.

Strong Terraform and GitLab CI/CD experience for infrastructure automation.

Hands‑on Datadog experience for monitoring, dashboards, and alerting.

Deep understanding of Kafka internals: partitions, replication, ISR, offsets, consumer groups, and delivery semantics.

Solid AWS fundamentals: VPC, security groups, IAM, KMS, CloudWatch, PrivateLink.

Strong Linux, networking, and scripting skills (Bash, Python).

Excellent communication and coordination skills to work across producer/consumer teams.

Preferred / Nice to Have

Experience with MirrorMaker 2, Confluent Replicator, or cluster linking for replication/migration.

Schema Registry, Kafka Connect, and ksqlDB experience.

Kafka Streams or Flink familiarity.

Kubernetes / EKS experience (e.g., Strimzi).

Certifications: Confluent Certified Administrator, AWS Solutions Architect/DevOps.

Experience in a regulated or large enterprise environment.

Droisys is an equal opportunity employer. We do not discriminate based on race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law. Droisys believes in diversity, inclusion, and belonging, and we are committed to fostering a diverse work environment.