Senior Kafka Engineer / Administrator Hybrid

Generis Tek Inc; Denver (CO); Híbrido; USD 55.000 - 62.000; Tempo integral

Resumo da vaga

Generis Tek Inc. seeks a senior Kafka engineer to own administration across on‑prem and AWS MSK in a hybrid setup. You will lead the migration, coordinate with producer and consumer teams, and ensure smooth cutovers with minimal downtime.

You will leverage Terraform and GitLab CI/CD to automate deployments, monitor performance with Datadog, and implement scalable, secure Kafka architectures in a hybrid cloud/on‑prem landscape.

Qualificações

• 5+ years in data/platform/infrastructure engineering with 4+ years operating Apache Kafka in production.
• Hands-on administration experience with Cloudera and Confluent Platform.
• Production experience with AWS MSK—provisioning, IAM, networking.
• Strong Terraform and GitLab CI/CD for automation.

Responsabilidades

• Own day‑to‑day Kafka administration across on‑prem and cloud.
• Drive migration from on‑prem (Cloudera/Confluent) to AWS MSK.
• Coordinate with producer/consumer teams for low‑risk cutover.
• Maintain security, monitoring, and performance of Kafka clusters.

Conhecimentos

Kafka administration

Cross-team coordination

Linux scripting

Communication

Problem solving

Ferramentas

Terraform

GitLab CI/CD

AWS MSK

Cloudera/Confluent

Datadog

Descrição da vaga

Location : Denver, CO - 80201 or St. Louis, MO - 63101 (Hybrid)

Project Duration : 12 months (Contract to Hire)

Pay Rate : $40 - $45 an hour on W2

Position Summary:

We are migrating a portfolio of Apache Kafka clusters from on-premises platforms (Cloudera and Confluent) to AWS MSK, managed through a CI/CD pipeline built on Terraform and GitLab. We are looking for a senior, hands‑on Kafka engineer to own day‑to‑day Kafka administration across both environments, drive the migration, and coordinate closely with producer and consumer application teams to ensure a smooth, low‑risk cutover. This is a senior individual‑contributor role for someone who has run Kafka at scale in production, is comfortable in a hybrid on‑prem/cloud world, and can act as the technical point of contact between platform and application teams.

Required Qualifications:

• 5+ years in data/platform/infrastructure engineering, with 4+ years’ operating Apache Kafka in production.
• Hands‑on administration experience with Cloudera and Confluent Platform.
• Production experience with AWS MSK (provisioning, configuration, IAM auth, networking).
• Proven experience migrating Kafka workloads to the cloud with minimal disruption.
• Strong Terraform and GitLab CI/CD experience for infrastructure automation.
• Hands‑on Datadog experience for monitoring, dashboards, and alerting.
• Deep understanding of Kafka internals: partitions, replication, ISR, offsets, consumer groups, and delivery semantics.
• Solid AWS fundamentals: VPC, security groups, IAM, KMS, CloudWatch, Private Link.
• Strong Linux, networking, and scripting skills (Bash, Python).
• Excellent communication and coordination skills to work across producer/consumer teams.

Preferred Skills / Nice to Have:

• Experience with Mirror Maker 2, Confluent Replicator, or cluster linking for replication/migration.
• Schema Registry, Kafka Connect, and ksq lDB experience.
• Kafka Streams or Flink familiarity.
• Kubernetes / EKS experience (e.g., Strimzi).
• Certifications: Confluent Certified Administrator, AWS Solutions Architect/DevOps.
• Experience in a regulated or large enterprise environment.

Mandatory Skills:

• Apache Kafka (Cloudera and Confluent)
• Kafka Administration (On-Prem + Cloud)
• Administer and operate Kafka clusters across Cloudera, Confluent Platform, and AWS MSK.
• Manage topics, partitions, replication, retention, quotas, ACLs, and consumer groups.
• Tune brokers, producers, and consumers for throughput, latency, and reliability.
• Handle patching, upgrades, capacity planning, and cluster health/performance troubleshooting.
• Configure and maintain security: TLS/SSL, SASL, MTLS, IAM (for MSK), RBAC/ACLs, and encryption at rest and in transit.
• Migration to AWS MSK
• Plan and execute migration of clusters and workloads from on‑prem (Cloudera/Confluent) to AWS MSK.
• Design cutover strategies (e.g., Mirror Maker 2 / Confluent Replicator / cluster linking) with minimal downtime and no data loss.
• Validate topic parity, offsets, throughput, and data integrity pre- and post‑migration.
• Define rollback plans and run migration dry‑runs in lower environments.
• CI/CD & Infrastructure as Code
• Build and maintain Kafka/MSK infrastructure using Terraform.
• Manage automated deployments and configuration through GitLab CI/CD pipelines.
• Implement Git Ops practices for topic, ACL, and cluster configuration management.
• Enforce version control, code review, and environment promotion (dev ? test ? prod).
• Observability & Monitoring
• Build and maintain monitoring, alerting, and dashboards in Datadog for Kafka/MSK (broker metrics, consumer lag, partition health, JVM/GC, throughput).
• Establish SLAs/SLOs and proactive alerting to reduce incidents and mean‑time‑to‑resolution.
• Correlate Kafka metrics with application and infrastructure telemetry.
• Producer / Consumer Coordination
• Serve as the primary technical liaison between the Kafka platform team and producer/consumer application teams.
• Advise application teams on client configuration, serialization/schema management, partitioning strategy, idempotence, and exactly‑once/at‑least‑once semantics.
• Support onboarding of new producers/consumers and troubleshoot client‑side issues (rebalancing, lag, poison messages).
• Coordinate migration schedules, testing, and cutover windows with each impacted team.