Senior Kafka Platform Engineer

Selby Jennings; Chicago (IL); On-site; USD 120,000 - 150,000; Full time

Job summary

A leading technology recruitment firm is seeking a Senior Kafka Platform Engineer to oversee the Kafka platform in Chicago. In this role, you will design, automate, and manage event-streaming capabilities, ensuring security and operational excellence. The ideal candidate has extensive Kafka expertise, a strong cloud-native mindset, and solid skills in automation and Kubernetes. This position offers the chance to work on modern deployment patterns in a dynamic environment.

Qualifications

• Extensive hands-on experience operating Kafka in production environments.
• Strong background in operating stateful systems on Kubernetes.
• Proficiency with Infrastructure as Code tools like Terraform.
• Solid scripting experience in Python, Go, or Java.
• Expertise in performance troubleshooting and capacity planning.

Responsibilities

• Own and operate production-grade Kafka clusters.
• Run Kafka on Kubernetes using Operators and Helm.
• Establish SLOs and maintain runbooks.
• Build and maintain monitoring for Kafka.
• Implement security measures for data-in-motion.

Skills

Kafka expertise

Kubernetes skills

Automation

Programming (Python, Go, Java)

Observability & Tuning

Security

Ecosystem components (Kafka Connect, Schema Registry)

Cloud knowledge (AWS, Azure, GCP)

Operational excellence

Tools

Terraform

Argo CD

GitOps

Prometheus

Grafana

OpenTelemetry

Job description

Overview

We're looking for a Senior Kafka Platform Engineer to design, automate, and scale a mission-critical event-streaming platform. In this role, you'll own the core Kafka environment—from brokers and storage through security, automation, and observability—while driving modern, Kubernetes-based deployment patterns. You'll build self-service tooling, define reliability standards, and collaborate closely with engineering teams to ensure robust, performant, and secure streaming capabilities. The ideal candidate brings deep Kafka expertise, strong automation skills, and a cloud-native engineering mindset.

Key Responsibilities

• Kafka Platform Ownership: Architect, deploy, and operate production-grade Kafka clusters (self-managed or cloud-hosted), overseeing upgrades, scaling strategies, capacity modeling, and multi-AZ/region resiliency.
• Kubernetes & Automation: Run Kafka on Kubernetes using Operators, Helm, and GitOps; build automation frameworks and guardrails using IaC to support repeatable, compliant, zero-downtime deployments.
• Ecosystem Services: Manage and optimize Kafka Connect, Schema Registry, and replication technologies (MirrorMaker 2, Cluster Linking); define connector standards and enable self-service provisioning.
• Reliability Engineering: Establish SLOs, own incident response, maintain runbooks, conduct postmortems, and develop automated remediation and resilience patterns.
• Observability: Build and maintain monitoring for metrics, logs, traces, consumer lag, partition health, and capacity insights using tools such as Prometheus, Grafana, Burrow, Cruise Control, or OpenTelemetry.
• Security & Compliance: Implement encryption, authentication, authorization, secrets management, network policies, and audit controls for secure data-in-motion.
• Streaming Best Practices: Guide application teams on topic strategy, partitioning, retention and compaction tuning, idempotency, ordering guarantees, schema evolution, DLQs, and exactly-once semantics.
• Cross-Functional Collaboration: Partner with application, data, platform, and SRE teams to provide tooling, documentation, enablement, and architectural guidance.
• Technical Leadership: Mentor engineers, help shape platform strategy, and contribute to long-term standards and roadmap decisions.

Core Skills

• Kafka Expertise: Extensive hands-on experience operating Kafka in production environments at scale, including brokers, controllers, replication, ISR dynamics, rebalancing, storage tiers, and failure recovery.
• Kubernetes Skills: Strong background operating stateful systems on Kubernetes using Operators, Helm, CRDs, and cloud-native patterns.
• Automation: Proficiency with IaC tools (e.g., Terraform), GitOps workflows (Argo CD or Flux), and CI/CD tooling for full lifecycle automation.
• Programming: Strong scripting and development experience in Python, Go, or Java; plus solid Bash and Linux fundamentals (networking, filesystems, JVM tuning).
• Observability & Tuning: Expertise in Kafka performance troubleshooting, capacity planning, monitoring stacks, and alerting workflows.
• Security: Hands-on experience with TLS/mTLS, SASL/OAuth, ACL/RBAC, and secret-management solutions such as Vault.
• Ecosystem Components: Experience with Kafka Connect, Schema Registry, MirrorMaker 2/Cluster Linking; familiarity with Cruise Control.
• Cloud: Knowledge of AWS, Azure, or GCP networking, IAM, and managed streaming services such as Confluent Cloud or AWS MSK.
• Operational Excellence: Demonstrated ability to write runbooks, lead incidents, and drive platform improvements.

Preferred Qualifications

• Experience with stream-processing frameworks (Kafka Streams, Flink, Spark Structured Streaming).
• Background running Strimzi or Confluent for Kubernetes in production.
• Knowledge of CDC technologies and connector operations at scale (e.g., Debezium).
• Experience designing multi-region architectures, cluster-linking strategies, and disaster-recovery processes.

Locations

• Chicago, IL
• New York, NY