Senior Confluent Kafka Administrator

Advance Career Solutions; Hyderabad, Pune District; Hybrid; INR 2,500,000 - 4,500,000; Full time

Job summary

Advance Career Solutions in Hyderabad/Pune is seeking a Senior Confluent Kafka Administrator / Streaming Engineer to own production Kafka platforms and real-time streaming solutions using Kafka Streams and Scala.

You will administer Kafka Connect, implement at-least-once/ exactly-once processing, and operate in a Kubernetes environment with Docker, Prometheus, Grafana and Confluent Control Center. Prior production experience is essential.

Qualifications

• Hands-on experience with Confluent Kafka administration in production environments.
• Strong knowledge of Kafka Streams, Scala and stream processing concepts.
• Experience with Kafka Connect, Schema Registry and data pipelines.
• Proficiency deploying and operating in Docker and Kubernetes clusters.
• Experience with monitoring using Prometheus and Grafana; ITIL/production support familiarity.

Responsibilities

• Manage and operate Confluent Kafka clusters for high availability and performance.
• Configure brokers, topics, partitions, retention and quotas; manage security (SSL/TLS, SASL, ACLs).
• Lead Kafka Connect deployments, connectors and DLQ handling in distributed mode.
• Develop streaming apps with Kafka Streams and Scala; implement error handling and retries.
• Support production SLAs, perform incident management and RCA activities.

Skills

Apache Kafka

Confluent Kafka

Kafka Administration

Kafka Streams

Scala

Kafka Connect

Schema Registry

Docker

Kubernetes

Prometheus

Grafana

Tools

Docker

Kubernetes

Prometheus

Grafana

Confluent Control Center

Job description

Role & responsibilities

Job Title: Senior Confluent Kafka Administrator / Streaming Engineer Scala
Experience: 6+ Years
Location: Hyderabad and Pune
Employment Type: Full Time
Industry: IT / Software Services

Job Summary

We are looking for a Senior Confluent Kafka Administrator / Streaming Engineer with strong hands-on experience in Confluent Kafka administration, Kafka Streams, Scala, Kafka Connect, Docker and Kubernetes.

The role involves managing production Kafka platforms, developing real-time streaming solutions using Kafka Streams and Scala, administering Kafka Connect, and supporting Kafka workloads in a Kubernetes-based production environment.

Key Responsibilities

Confluent Kafka Administration

• Manage and operate Confluent Kafka clusters for high availability, stability and performance.
• Configure and optimize brokers, topics, partitions, replication, retention, quotas and consumer groups.
• Manage Kafka security including SSL/TLS, SASL, ACLs, RBAC, secrets and certificate rotation.
• Perform Kafka upgrades, patching, capacity planning and disaster recovery activities.
• Work with MirrorMaker 2 / Replicator for replication and DR requirements.
• Troubleshoot Kafka latency, throughput, consumer lag, ISR, disk, network and rebalancing issues.
• Perform root cause analysis and implement permanent fixes.

Kafka Streams & Scala Development

• Design and develop real-time streaming applications using Kafka Streams API and Scala.
• Implement transformations, joins, windowing, state stores and stream processing.
• Optimize applications for scalability, performance and reliability.
• Implement at-least-once / exactly-once processing, error handling, retries and DLQ patterns.
• Integrate streaming applications with downstream systems and applications.

Kafka Connect

• Build, configure and administer Kafka Connect in distributed mode.
• Manage source/sink connectors, converters and SMTs.
• Handle connector deployment, scaling, monitoring, tuning and troubleshooting.
• Manage connector failures, recovery and Dead Letter Queue (DLQ) mechanisms.

Kubernetes / DevOps / Monitoring

• Deploy and support Kafka-related services in Docker and Kubernetes environments.
• Work with Kubernetes manifests, Helm, probes, configurations, secrets and rollouts.
• Implement monitoring and observability using Prometheus, Grafana and Confluent Control Center.
• Automate operational activities using scripts and tools.
• Contribute to CI/CD pipelines and release processes.

Production Support / ITIL

• Handle production incidents, changes and problems using ServiceNow.
• Participate in INC / CHG / PRB management and CAB processes.
• Prepare change plans, risk/impact assessments, implementation steps, validation and rollback plans.
• Participate in incident calls, RCA and post-incident activities.
• Maintain Kafka architecture documents, runbooks, SOPs and operational procedures.

Mandatory Skills

• Apache Kafka / Confluent Kafka
• Kafka Administration
• Kafka Streams
• Scala
• Kafka Connect
• Schema Registry
• Docker
• Kubernetes
• Prometheus / Grafana
• Strong knowledge of distributed systems and event-driven architecture
• Production support and troubleshooting experience

Good to Have

• ksqlDB / KSQL
• Confluent Control Center
• MirrorMaker 2 / Replicator
• CI/CD and DevOps automation
• ServiceNow / ITIL
• Kafka security: SSL/TLS, SASL, ACL, RBAC
• Experience with high-volume, low-latency real-time streaming platforms

Ideal Candidate

Candidates should have strong hands-on experience in Confluent Kafka administration + Kafka Streams development using Scala. Experience should include production Kafka operations along with Kafka Connect, Kubernetes and real-time streaming.

Mandatory Skill: Kafka