Senior Kafka Engineer: Cloud Migrations to MSK

Droisys; St. Louis (MO); Vor Ort; USD 171.924.000 - 186.252.000; Vollzeit

Zusammenfassung

Droisys is migrating a portfolio of Apache Kafka clusters from on-premises platforms (Cloudera and Confluent) to AWS MSK, managed via Terraform and GitLab CI/CD. We seek a senior, hands-on Kafka engineer to own day-to-day administration across environments, drive the migration, and coordinate with producer/consumer teams to ensure a smooth cutover.

This is a senior IC role for someone who has run Kafka at scale in production, in a hybrid on-prem/cloud setup, and who can be the technical contact

Qualifikationen

• 5+ years in data/platform/infrastructure engineering.
• 4+ years operating Apache Kafka in production.
• Hands-on administration with Cloudera and Confluent Platform.
• Production experience with AWS MSK (IAM auth, networking).
• Experience migrating Kafka workloads to the cloud with minimal disruption.
• Terraform and GitLab CI/CD for automation.
• Datadog for monitoring, dashboards, and alerting.
• Deep understanding of Kafka internals and delivery semantics.
• Solid AWS fundamentals and Linux scripting.
• Strong communication across producer/consumer teams.

Aufgaben

• Kafka Administration (On‑Prem + Cloud): manage clusters, topics, partitions, replication, quotas, ACLs, and consumer groups.
• Tune brokers, producers, and consumers for throughput and reliability.
• Handle patching, upgrades, capacity planning, and health checks.
• Configure security: TLS/SSL, SASL, IAM, RBAC, encryption.
• Migration to AWS MSK: cutover planning, minimal downtime, data integrity checks.
• CI/CD & IaC: Terraform and GitLab pipelines for deployments.
• Observability: Datadog dashboards, alerts, and SLA tracking.
• Producer/Consumer coordination: liaison between platform and applications.
• Guide client configuration, partitioning, and exactly-once semantics.
• Coordinate migration schedules and testing with teams.

Kenntnisse

Kafka administration

Terraform

GitLab CI/CD

Datadog monitoring

AWS MSK

Linux scripting

Networking basics

Communication

Production Kafka

Cluster tuning

Tools

Cloudera

Confluent Platform

AWS MSK

Kubernetes

Jobbeschreibung

Droisys is migrating a portfolio of Apache Kafka clusters from on-premises platforms (Cloudera and Confluent) to AWS MSK, managed via Terraform and GitLab CI/CD. We seek a senior, hands-on Kafka engineer to own day-to-day administration across environments, drive the migration, and coordinate with producer/consumer teams to ensure a smooth cutover.

This is a senior IC role for someone who has run Kafka at scale in production, in a hybrid on-prem/cloud setup, and who can be the technical contact