Kafka developer

Avance Consulting; Sheffield; On-site; GBP 70,000 - 100,000; Full time

Job summary

A leading technology consulting firm is seeking a highly skilled Senior Kafka Engineer to enhance their Kafka architecture. In this role, you'll design, implement, and manage Kafka clusters while ensuring systems' integrity and performance. The position demands strong expertise in upgrading Kafka, developing monitoring tools, and collaborating with engineering teams. Ideal candidates will have 8+ years of experience with Kafka and proficiency in cloud-based deployments.

Qualifications

• 8+ years of experience working with Apache Kafka.
• Proven experience upgrading Kafka clusters, especially from version 3.3.1.
• Strong understanding of Kafka architecture and deployment.

Responsibilities

• Design, implement, and maintain Kafka clusters.
• Lead upgrades of existing Kafka clusters.
• Mentor junior engineers.

Skills

Kafka architecture

Monitoring and alerting systems

Troubleshooting

Collaboration

Problem-solving

Tools

Kafka Connect

Kafka Zookeeper

Kubernetes

Kafka Streams

ksqlDB

AWS

Azure

GCP

Job description

We are seeking a highly experienced and skilled Senior Kafka Engineer to join our team. The ideal candidate will have a proven track record of designing, implementing, and maintaining large-scale Kafka deployments, with specific experience upgrading Kafka clusters. This role requires a deep understanding of Kafka's architecture, internals, and best practices. The candidate must be comfortable working independently and collaboratively within a fast-paced environment.

Responsibilities:

Design, implement, and maintain high-throughput, low-latency Kafka clusters.

Lead the upgrade of existing Kafka clusters from version 3.3.1 to the latest supported version, ensuring minimal downtime and data integrity.

Experience in confluent migration (link removed)>

Develop and implement monitoring and alerting systems for Kafka.

Troubleshoot and resolve Kafka-related issues.

Participate in on-call rotation to ensure 24/7 system availability.

Collaborate with other engineers to integrate Kafka into existing systems.

Design and implement Kafka security best practices.

Contribute to the improvement of our Kafka infrastructure.

Mentor junior engineers and share expertise.

Stay current with the latest Kafka technologies and best practices.

Qualifications:

8+ years of experience working with Apache Kafka.

Proven experience upgrading Kafka clusters, specifically from version 3.3.1 to a current supported version. Detail of the specific versions involved is highly desirable.

Strong understanding of Kafka architecture, including topics, partitions, consumers, and producers.

Experience with Kafka Connect.

Experience with Kafka Zookeeper, Kraft.

Experience working in Confluent Kafka.

Experience in deploying the Kafka image in Kubernetes.

Experience in setting up Stretched Kafka Cluster.

Experience with Kafka Streams or ksqlDB a plus.

Proficiency in at least one scripting language (e.g., Python, Shell).

Experience with cloud-based Kafka deployments (AWS, Azure, GCP) is highly preferred.

Experience with Kafka monitoring and alerting tools.

Excellent problem-solving and troubleshooting skills.

Strong communication and collaboration skills.

* Free services are subject to limitations