Kafka Administrator

Wenger & Watson; India; Hybrid; INR 1,400,000 - 2,200,000; Full time

Job summary

Wenger & Watson in India seeks a senior Kafka/Confluent administrator to design, deploy, and operate large-scale streaming pipelines. You will manage clusters, ensure high availability across data centers, and tune performance for low latency.

The role covers security, authentication, authorization, backups, upgrades with minimal downtime, and ongoing governance of topics, schemas, and connectors using Terraform, Helm, and GitOps workflows.

Qualifications

• 5+ years in platform/SRE/DevOps with production Kafka/Confluent admin
• Hands-on with Kafka brokers, partitions, replication, and log management
• Experience with Confluent Platform components or Confluent Cloud administration
• Proficiency with Kafka CLI, AdminClient, ACLs, quotas, and exactly-once semantics
• Familiarity with Prometheus/Grafana, JMX exporters, and alerting systems
• Linux, JVM tuning, and network tuning including TCP buffers and MTU

Responsibilities

• Operate and scale Kafka Connect with distributed workers
• Manage authentication (mTLS/SASL) and authorization (RBAC, ACLs)
• Plan upgrades/patching with minimal downtime
• Maintain backup/restore workflows for metadata, Schema Registry, and Connect configs
• Enforce encryption in transit and at rest, secret management (Vault/AKV)
• Define topic naming conventions, retention policies, and schema governance
• Coordinate vulnerability management and security hardening
• Deploy metrics/logging pipelines (ELK/OpenSearch) and tracing (OpenTelemetry)
• Monitor SLOs/SLIs and optimize broker performance
• Size clusters and plan capacity for multiAZ/multiregion deployment

Skills

Kafka admin

Confluent Platform

JVM tuning

Terraform

Kubernetes

Prometheus/Grafana

Security and encryption

Networking

Tools

Terraform

Helm

GitOps

Job description

• Exp- 5-12 Yrs
• Provision, configure, and manage Kafka clusters (Apache Kafka KRaft or ZooKeeper-backed if legacy), Confluent Platform components (Schema Registry, Kafka Connect, ksqlDB, REST Proxy, Control Center), and Confluent Cloud resources.
• Ensure high availability (HA) and resilience across multiAZ/region deployments; manage partition replication, min.insync.replicas (ISR), and rack awareness.
• Implement and maintain client quotas, broker quotas, throttling, and backpressure strategies.
• Plan and execute upgrades/patching with zero/minimal downtime; maintain version currency against EOL and security advisories.
• Own backup/restore workflows for metadata (e.g., cluster configs), Schema Registry, and Connect configs
• Implement authentication (mTLS/SASL: SCRAM, OAUTHBEARER with IdP, or Kerberos if legacy) and authorization (Kafka ACLs, Confluent RBAC).
• Enforce encryption in transit and at rest, secret management (Vault/AKV/SSM), secure endpoint exposure, and private connectivity (VPC peering/PrivateLink).
• Govern topic naming conventions, retention policies, schema evolution rules (backward/fully compatible), and PII handling.
• Maintain audit trails (broker, Schema Registry, Control Center, Confluent Cloud audit logs) and compliance artifacts (SOX, GDPR, HIPAA/PCI as applicable).
• Coordinate vulnerability management (CVE watch, hardening baselines, CIS/STIG alignment).
• Deploy and maintain metrics pipelines (JMX Prometheus/Grafana), logs (ELK/OPensearch), and tracing (OpenTelemetry where applicable).
• Define and monitor SLOs/SLIs (produce/consume latency, endtoend lag, broker CPU/heap/file handles, network throughput, GC pauses).
• Analyze broker hotspots, partition skew, large message handling, compaction and retention settings, and tune GC/JVM for sustained throughput.
• Implement consumer lag monitoring and alerting with actionable runbooks to prevent data loss or SLA breaches.
• Model workload growth, topic/partition scaling, storage/IOPS, network egress/ingress, and broker sizing.
• For Confluent Cloud, optimize environment cluster topic hierarchy, throughput tiers, retention costs, and data flow egress; rightsize Dedicated vs. Standard clusters.
• Design partitioning strategies to meet throughput targets while balancing consumer concurrency and avoiding small partitions antipatterns
• Automate cluster and topic lifecycle using Terraform (providers for Kafka/Confluent), Helm, GitOps workflows.
• Operate and scale Kafka Connect with distributed workers; manage connectors (JDBC, Debezium CDC, S3/ADLS/GCS, Elasticsearch, etc.), transforms (SMTs), and converter settings.
• Administer Schema Registry (compatibility levels, subject naming strategies, serializers/deserializers, schema size & references).
• Operate ksqlDB and/or Flink for streaming SQL; enforce resource limits and multitenant governance.
• Engineer lowlatency and secure connectivity across data centers/VPCs/VNETs; manage DNS, TLS SANs, LB configurations, advertised.listeners, and intercluster networking.

Required Skills & Experience

• 510 years in platform/SRE/DevOps; 5+ years dedicated to Kafka/Confluent admin in production.
• Strong handson with Apache Kafka internals: brokers, controllers (KRaft), partitions/replication, log segments, compaction/retention.
• Confluent Platform: Schema Registry, Kafka Connect, ksqlDB, Control Center, Confluent REST APIs; or Confluent Cloud admin (environments, clusters, RBAC, audit logs, networking).
• Proficiency with Kafka CLI tools and AdminClient usage; deep knowledge of ACLs, quotas, transactions, idempotent producers, and exactlyonce semantics scenarios.
• Prometheus/Grafana, JMX exporters, alerting systems (Alertmanager etc).
• JVM tuning, Linux performance (NUMA, file descriptors, page cache, disk/RAID, XFS), and network tuning (TCP buffers, MTU, jumbo frames where applicable).
• Linux administration (systemd, journald, kernel params), virtualization/containers (Docker, Kubernetes).
• Cloud networking (VPC/VNET peering, transit gateways, PrivateLink/Private Service Connect), multiregion architectures.

Preferrable -

• Confluent Certified Administrator for Apache Kafka (CCAAK), CKA/CKAD, HashiCorp Terraform Associate, major cloud certs (Azure/AWS/GCP).
• Experience with Confluent for Kubernetes (CFK) for operatorbased deployments.