Kafka Architect

Capgemini; Leeds; On-site; GBP 60,000 - 100,000; Full time

Job summary

An established industry player is seeking a Kafka Architect to design and implement scalable, real-time data processing systems. This exciting role involves architecting Kafka clusters, ensuring high availability, and integrating with various data processing tools. The ideal candidate will possess extensive experience with Apache Kafka, Kafka Streams, and cloud platforms. You will collaborate with cross-functional teams to meet business needs and implement security measures for data protection. Join a forward-thinking organization where your expertise will drive innovative solutions in real-time data processing.

Qualifications

• Extensive experience with Apache Kafka and real-time architecture.
• Strong knowledge of Kafka Streams, Connect, and cloud platforms.

Responsibilities

• Design scalable, real-time systems in Kafka and ensure high availability.
• Monitor Kafka performance and troubleshoot issues effectively.

Skills

Apache Kafka

Kafka Streams

Kafka Connect

Spark Streaming

Flink

Beam

Problem-solving skills

Tools

GCP Pub/Sub

Job description

Job Description

Job Title: Kafka Architect

Your Role

A Kafka Real-Time Architect is responsible for designing and implementing scalable, real-time data processing systems in Kafka. This role involves architecting Kafka clusters, ensuring high availability, and integrating with other data processing tools and platforms. As part of the CTO Data Ingestion Service, the incumbent will be required to:

• Design and architect scalable, real-time systems in Kafka.
• Configure, deploy, and maintain Kafka clusters to ensure high availability and scalability.
• Integrate Kafka with other data processing tools and platforms such as Kafka Streams, Kafka Connect, Spark Streaming, Schema Registry, Flink, and Beam.
• Collaborate with cross-functional teams to understand data requirements and design solutions that meet business needs.
• Implement security measures to protect Kafka clusters and data streams.
• Monitor Kafka performance and troubleshoot issues to ensure optimal performance. Provide technical guidance and support to development operations teams.
• Stay updated with the latest Kafka features, updates, and industry practices.
• Have extensive experience with Apache Kafka and real-time architecture including event-driven frameworks.
• Possess strong knowledge of Kafka Streams, Kafka Connect, Spark Streaming, Schema Registry, Flink, and Beam.
• Have experience with cloud platforms such as GCP Pub/Sub.
• Exhibit excellent problem-solving skills.

Your Profile

• Knowledge of Kafka data pipelines and messaging solutions to support critical business operations and enable real-time data processing.
• Experience in monitoring Kafka performance, enhancing decision-making and operational efficiency.
• Ability to collaborate with development teams to integrate Kafka applications and services.
• Maintain an architectural library for Kafka deployment models and patterns.
• Assist developers in maintaining Kafka connectors such as JDBC, MongoDB, and S3 connectors, along with topics schemas, to streamline data ingestion from databases, NoSQL data stores, and cloud storage, enabling faster data insight.