Apache Kafka Engineer

Stott and May; England; On-site; GBP 70,000 - 100,000; Full time

Job summary

A leading company is seeking a Senior Apache Kafka Engineer to lead the design, development, and management of an enterprise event streaming platform. This position requires in-depth Kafka expertise and strong system design skills, along with experience in managing complex Kafka environments and deploying real-time data pipelines across multiple environments.

Qualifications

• 5+ years in software/data engineering, including 3+ years hands-on with Kafka.
• Experience managing complex Kafka environments.
• Strong understanding of Kafka internals and cloud integrations.

Responsibilities

• Own and evolve Kafka infrastructure: optimize architecture.
• Design and implement scalable event-driven systems.
• Develop and maintain Kafka clusters and integration systems.

Skills

Kafka expertise

System design

Java

Scala

Python

Docker

Kubernetes

Terraform

Helm

Tools

Kafka Streams

ksqlDB

Prometheus

Grafana

Datadog

Job description

Apache Kafka Engineer

My client is looking for an Senior Apache Kafka Engineer to lead the design, development, and management of our enterprise event streaming platform. This role requires deep Kafka expertise, strong system design skills, and hands-on experience managing large-scale, production-grade deployments.

Key Responsibilities

• Own and evolve a critical Kafka infrastructure: assess, stabilize, and optimize architecture.

• Design and implement scalable, event-driven systems across environments (dev, staging, prod).

• Develop and maintain Kafka clusters, topics, partitions, schemas (Avro), and connectors.

• Integrate Kafka with external systems and ensure reliability, security, and observability.

• Troubleshoot delivery issues, latency, consumer lag, and performance bottlenecks.

• Drive documentation, training, incident resolution, and continuous improvement.

Qualifications

• 5+ years in software/data engineering, with 3+ years of hands-on Kafka experience.

• Proven track record managing complex Kafka environments.

• Strong understanding of Kafka internals (brokers, replication, KRaft/ZooKeeper, ISR).

• Experience with Kafka Streams, ksqlDB, and building real-time data pipelines.

• Proficient in Java, Scala, or Python.

• Familiarity with Kafka Connect, Schema Registry, and common integrations (DBs, Elasticsearch, cloud).

• Experience with Docker, Kubernetes, Terraform, and Helm.

• Cloud Kafka experience (AWS MSK, Confluent, Azure Event Hubs).

• Skilled with monitoring tools (Prometheus, Grafana, Datadog, etc.).