Kafka Architect- Cloudera

JR United Kingdom; Bradford; On-site; GBP 60,000 - 90,000; Full time

Job summary

A leading company is seeking an experienced Kafka Architect to design and implement scalable data streaming solutions. You'll be responsible for creating architectures, optimizing Kafka clusters, and collaborating with cross-functional teams to support real-time data needs. Ideal candidates will have a solid background in distributed systems and hands-on experience with Apache Kafka.

Qualifications

• Strong hands-on experience with Apache Kafka including Kafka Connect, Kafka Streams.
• Experience with Cloudera environment for Kafka.
• Understanding of data serialization formats like Avro, JSON, Protobuf.

Responsibilities

• Design scalable architectures using Apache Kafka on Cloudera.
• Lead the configuration and optimization of Kafka clusters.
• Integrate Kafka with various data sources and monitor performance.

Skills

Apache Kafka

Java

Scala

Python

Data Processing

Prometheus

Grafana

CI/CD

Containerization

Job description

Social network you want to login/join with:

We are looking for an experienced Kafka Architect to design and implement scalable, high-throughput data streaming solutions using Apache Kafka on Cloudera (on-premises). The ideal candidate will have a strong background in distributed systems, data pipelines, and real-time data processing.

Key Responsibilities:

-Design and implement scalable Kafka-based architectures using open-source -Kafka on Cloudera on-premises infrastructure.

-Lead the setup, configuration, and optimization of Kafka clusters.

-Define standards and best practices for Kafka producers, consumers, and streaming applications.

-Integrate Kafka with various data sources, storage systems, and enterprise applications.

-Monitor Kafka performance and ensure high availability, fault tolerance, and data security.

-Collaborate with DevOps, Data Engineering, and Application teams to support real-time data needs.

-Automate deployment and configuration using tools like Ansible, Terraform, or Cloudera Manager.

-Provide technical leadership and mentorship to junior team members.

Required Skills:

-Strong hands-on experience with Apache Kafka (including Kafka Connect, Kafka Streams).

-Experience with Cloudera distribution for Kafka on on-premises environments.

-Solid knowledge of Kafka internals – topics, partitions, consumer groups, offset management, etc.

-Experience with data serialization formats like Avro, JSON, Protobuf.

-Proficient in Java, Scala, or Python for Kafka-based development.

-Familiarity with monitoring tools (Prometheus, Grafana, Confluent Control Center, etc.).

-Understanding of networking, security (SSL/SASL), and data governance.

-Experience with CI/CD pipelines and containerization (Docker, Kubernetes) is a plus.