Kafka Architect- Cloudera

Vallum Associates; Leeds; On-site; GBP 70,000 - 100,000; Full time

Job summary

A leading technology company seeks an experienced Kafka Architect to design and implement high-throughput data streaming solutions using Apache Kafka on Cloudera. The successful candidate will play a pivotal role in optimizing Kafka clusters and integrating with various data systems while providing technical leadership.

Qualifications

• Extensive hands-on experience with Apache Kafka.
• Experience with Cloudera distribution in on-premises setups.
• Proficient in creating high-volume, low-latency data pipelines.

Responsibilities

• Design and implement scalable Kafka-based architectures.
• Lead setup and optimization of Kafka clusters.
• Provide technical leadership and mentorship to junior members.

Skills

Apache Kafka

Java

Scala

Python

Data Serialization Formats

Real-time Data Processing

Data Pipelines

Monitoring Tools

Tools

Ansible

Terraform

Docker

Kubernetes

Job description

Job description

Job Summary:

We are looking for an experienced Kafka Architect to design and implement scalable, high-throughput data streaming solutions using Apache Kafka on Cloudera (on-premises). The ideal candidate will have a strong background in distributed systems, data pipelines, and real-time data processing.

Key Responsibilities:

-Design and implement scalable Kafka-based architectures using open-source -Kafka on Cloudera on-premises infrastructure.

-Lead the setup, configuration, and optimization of Kafka clusters.

-Define standards and best practices for Kafka producers, consumers, and streaming applications.

-Integrate Kafka with various data sources, storage systems, and enterprise applications.

-Monitor Kafka performance and ensure high availability, fault tolerance, and data security.

-Collaborate with DevOps, Data Engineering, and Application teams to support real-time data needs.

-Automate deployment and configuration using tools like Ansible, Terraform, or Cloudera Manager.

-Provide technical leadership and mentorship to junior team members.

Required Skills:

-Strong hands-on experience with Apache Kafka (including Kafka Connect, Kafka Streams).

-Experience with Cloudera distribution for Kafka on on-premises environments.

-Proficiency in designing high-volume, low-latency data pipelines.

-Solid knowledge of Kafka internals – topics, partitions, consumer groups, offset management, etc.

-Experience with data serialization formats like Avro, JSON, Protobuf.

-Proficient in Java, Scala, or Python for Kafka-based development.

-Familiarity with monitoring tools (Prometheus, Grafana, Confluent Control Center, etc.).

-Understanding of networking, security (SSL/SASL), and data governance.

-Experience with CI/CD pipelines and containerization (Docker, Kubernetes) is a plus.