Technical Lead Confluent Kafka

Cerebraix; Chennai District; Sur place; INR 4 000 000 - 6 000 000; Plein temps

Résumé du poste

Unknown company is seeking a Technical Lead for Confluent Kafka with 13+ years experience. You will design and develop Kafka-based applications, adaptors, and connectors, while optimizing performance on cluster nodes.

You will implement security models, prepare production promotions, and lead testing and go-live activities with the team.

Qualifications

• Proficient in analyzing business requirements for streaming pipelines.
• Designed retry topics, DLQs, idempotent processing and audit trails.
• Managed compatibility modes and schema evolution for schemas.
• Guided Confluent Platform upgrades and migrations.
• Escalation point for rebalance storms and cross-pipeline issues.
• Authored low-level design documents and maintained them.
• Led code reviews and mentored developers.

Responsabilités

• Design and develop Confluent Kafka-based applications and connectors.
• Tune performance of custom adaptors and connectors on cluster nodes.
• Implement security models (RBAC, ACLs, encryption, mTLS/SSL).
• Write Confluent scripts for production promotion of topics and connectors.
• Monitor, troubleshoot and fix system defects in production.
• Prepare production deployments and coordinate with testers for readiness.
• Support UT, SIT, UAT, Staging and Go-Live activities.

Connaissances

Event streaming requirement analysis

Audit and retry mechanisms

Schema Registry governance

Cluster upgrade experience

Complex troubleshooting

LLD documentation ownership

Technical leadership

Description du poste

Technical Lead - Confluent Kafka (13+ years)

Experience gate: 13+ years total, with 6+ years hands-on on Kafka/Confluent Platform and at least 2-3 years leading a development team on Kafka/streaming work.

Responsibilities:

• Core Development: Design and develop Confluent Kafka-based applications, custom adaptors, custom connectors, KSQL scripts, and KStream applications.
• Performance Tuning: Execute performance optimization and fine-tuning of custom adaptors and connectors running on Confluent cluster nodes.
• Security Implementation: Implement security architecture models (such as RBAC, ACLs, Encryption, and mTLS/SSL) directly within the applications and connectors.
• Deployment Scripting: Write and prepare Confluent scripts for promoting changes related to Topics, Connectors, Streams, Adaptors, and KSQL into production environments.
• Troubleshooting & Bug Fixing: Perform hands-on monitoring, troubleshooting, and bug fixing to resolve system-related development defects.
• Production Readiness: Prepare and package production deployments, aligning with standard schedules and coordinating with testers for readiness validation.
• Testing & Go-Live: Provide direct, hands-on support during Unit Testing (UT), System Integration Testing (SIT), User Acceptance Testing (UAT), Staging, and Go-Live events.

Mandatory skills:

1. Event streaming requirement analysis and design. Has turned business requirements into stream pipeline designs, covering topic modelling, partitioning and keying strategy, data contracts and event-driven architecture patterns.
2. Audit and retry mechanisms, end to end. Has designed and owned retry topics, DLQs, idempotent processing, replay and reprocessing, and audit trails in production. This is called out explicitly in the JD and is a strong filter.
3. Schema Registry governance. Has set and enforced compatibility modes (BACKWARD, FORWARD, FULL and their transitive variants) and schema evolution rules. Has managed subject naming strategies and built compatibility checks into CI pipelines.
4. Cluster upgrade experience. Has provided development-side guidance on Confluent Platform minor and major upgrades. That includes client and broker compatibility, rolling upgrades, rollback testing in staging, and ideally ZooKeeper-to-KRaft migration exposure.
5. Complex and systemic troubleshooting. Has been the escalation point for issues like rebalance storms, under-replicated partitions or ISR shrink, ordering and duplication problems, and cross-pipeline failures.
6. LLD documentation ownership. Has authored and maintained Low-Level Design documents in Confluence (Atlassian).
7. Technical leadership. Has led code reviews, enforced coding standards, mentored developers, and owned team deliverables through UT, SIT, UAT and Go-Live.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.