Confluent Kafka Tech Lead Required with Experience 9 - 18 years.

Lightup Technologies; Dadri, Chennai District, Bengaluru; Vor Ort; INR 2.500.000 - 4.500.000; Vollzeit

Zusammenfassung

Lightup Technologies in India seeks a Technical Lead for Confluent Kafka to spearhead development within the Kafka pod. You will guide complex implementations, enforce governance, and ensure robust end-to-end stream pipelines.

You will drive design, security, and upgrade planning while mentoring the team and owning critical LL documentation in Confluence.

Qualifikationen

• 9+ years total experience in software development with 6+ years on Kafka/Confluent Platform
• 3+ years leading a development team on Kafka/streaming projects
• Hands-on with Java Producer/Consumer APIs and Kafka security配置

Aufgaben

• Act as senior technical voice and bridge requirements to execution
• Guide team through complex Kafka implementations and governance
• Evaluate and lead Confluent streaming requirements for new pipelines
• Collaborate with cross-functional teams on event streaming use cases
• Develop and maintain audit and retry mechanisms in Confluent Kafka apps
• Enforce Schema Registry governance and compatibility policies in code
• Own LL design documentation in Confluence
• Provide guidance during cluster upgrades and rollbacks in staging
• Lead bug resolution for complex, systemic issues
• Support UT/SIT/UAT/Staging/Go-Live testing phases

Kenntnisse

Java (Core + Kafka client APIs)

Kafka Streams

ksqlDB / KSQL

Kafka Connect

Performance tuning

Security at app level (SASL/OAuth/OIDC

Schema Registry governance

CI/CD tooling (Jenkins/GitLab/Azure)

Monitoring & troubleshooting (Control 

Event streaming design & architecture

Tools

Confluent CLI

Confluent Control Center

Schema Registry

KSQL tooling

Testcontainers/Embedded Kafka

Jobbeschreibung

Job Description

Technical Lead Confluent Kafka

Role Overview

The Technical Lead Confluent Kafka will spearhead the development efforts within the Kafka pod.

Key Responsibilities

• You will act as the senior technical voice for the development team, bridging the gap between requirements and execution.
• You will guide the development team through complex implementations, enforce code-level governance, and ensure that stream pipelines are robust, secure, and well-documented.
• Requirement Evaluation & Collaboration: Lead the evaluation of Confluent streaming requirements for new or modified stream pipelines.
• Collaborate with cross-functional teams to provide critical developer inputs for event streaming use cases.
• Advanced Development & Mechanisms: Drive the development and maintenance of complex, endto- end audit and retry mechanisms within Confluent Kafka-based applications.
• Code-Level Governance: Enforce Schema Registry governance, schema evolution, and compatibility policies directly within the codebase.
• Documentation Ownership: Take ownership of the development-side documentation, specifically driving the creation and maintenance of detailed Low-Level Design (LLD) documents in Confluence.
• Upgrade & Testing Oversight: Provide senior-level development guidance during minor and major Cluster Upgrade planning, execution, and rollback testing in staging environments.
• Complex Troubleshooting: Lead the technical resolution and bug fixing for complex, systemic issues escalated from the core development team.
• Testing Support: Oversee and support the development team's deliverables during UT, SIT, UAT, Staging, and Go-Live phases.

Experience gate

9+ years total, with 6+ years hands-on on Kafka/Confluent Platform and at least 3 years leading a development team on Kafka/streaming work.

Mandatory skills

• Java (Core + Kafka client APIs). Strong Java with hands-on Producer/Consumer API work. The candidate should understand idempotent producers, transactions, exactly-once vs at-least-once semantics, offset management and consumer groups. Spring Kafka is a plus.
• Kafka Streams (KStream/KTable). Has built and run stream applications in production, covering stateful operations, joins, windowing, state stores and interactive queries.
• ksqlDB / KSQL. Has written streams, tables and persistent queries, and promoted them across environments.
• Kafka Connect, including custom connector development. Has built custom Source/Sink connectors with the Connect API and written Single Message Transforms (SMTs). Has also configured and deployed out-of-the-box connectors.
• Performance tuning. Has hands-on tuned producer, consumer and connector configurations (batching, linger, compression, fetch and poll settings) and partition and key strategy.
• Has diagnosed consumer lag and throughput or latency issues on Confluent clusters.
• Security implementation at application level. Has configured SASL/SCRAM or OAuth, SSL/mTLS, ACLs and Confluent RBAC in client applications and connectors, plus encryption of data in transit and at rest.
• Schema Registry and serialization. Has used Avro and/or Protobuf/JSON Schema with Confluent Schema Registry in producers, consumers and connectors.
• Deployment scripting and environment promotion. Has scripted promotion of topics, connectors, streams and ksqlDB queries to production using shell and/or Python, the Confluent CLI and the Connect/ksqlDB REST APIs. Has worked with at least one CI/CD tool, such as Jenkins, GitLab CI or Azure DevOps.
• Monitoring and troubleshooting. Has used Confluent Control Center, JMX metrics and logs to debug rebalances, lag, poison-pill messages and dead letter queues (DLQs).
• Testing and release support. Has written unit tests for Kafka code (TopologyTestDriver, Embedded Kafka or Testcontainers) and has supported SIT, UAT, Staging and Go-Live releases. Comfortable on Linux
• Event streaming requirement analysis and design. Has turned business requirements into stream pipeline designs, covering topic modelling, partitioning and keying strategy, data contracts and event-driven architecture patterns
• Audit and retry mechanisms, end to end. Has designed and owned retry topics, DLQs, idempotent processing, replay and reprocessing, and audit trails in production. This is called out explicitly in the JD and is a strong filter
• Schema Registry governance. Has set and enforced compatibility modes (BACKWARD, FORWARD, FULL and their transitive variants) and schema evolution rules. Has managed subject naming strategies and built compatibility checks into CI pipelines
• Cluster upgrade experience. Has provided development-side guidance on Confluent Platform minor and major upgrades. That includes client and broker compatibility, rolling upgrades, rollback testing in staging, and ideally ZooKeeper-to-KRaft migration exposure
• Complex and systemic troubleshooting. Has been the escalation point for issues like rebalance storms, under-replicated partitions or ISR shrink, ordering and duplication problems, and cross-pipeline failures
• LLD documentation ownership. Has authored and maintained Low-Level Design documents in Confluence (Atlassian)
• Technical leadership. Has led code reviews, enforced coding standards, mentored developers, and owned team deliverables through UT, SIT, UAT and Go-Live.

Mandatory Requirements to avoid screen rejection

• All employments with Start and End dates
• All education with Start and End Dates
• Linkedin profile link
• Valid passport end date
• Valid PAN card
• Mobile Number
• E-mail ID
• Preferred job location

Note: Please mention all above in your resume.