Streaming Data Engineer- Bangalore

Celebal Technologies; Bengaluru; In loco; INR 4.000.000 - 7.000.000; Tempo pieno

Descrizione del lavoro

Celebal Technologies in Bengaluru seeks a Senior Streaming Data Engineer to design, operate, and migrate real-time data platforms on Confluent Platform and Confluent Cloud. You will own critical pipelines moving data between applications, databases, and analytics systems, and lead migrations into and out of Confluent.

Experience with other streaming platforms (Kinesis, Pub/Sub, Event Hubs, Pulsar) is valued, enabling cross-skill and mapping new platforms onto Kafka.

Competenze

• 5+ years in data/platform/distributed systems engineering with 3+ years on Kafka/Confluent in production.
• Deep Kafka internals: partitions, offsets, exactly-once semantics.
• Proficiency in Java or Python, strong SQL, and shell scripting.
• Experience with AWS or GCP networking, IAM and storage.
• CI/CD tooling and monitoring experience.

Mansioni

• Design and manage Kafka solutions on Confluent including topic design and quotas.
• Lead migrations into/out of Confluent using MirrorMaker 2, Replicator, and connectors.
• Ensure ordering, idempotency, and offset continuity during cutovers.
• Build/Review producers, consumers, and Kafka Connect pipelines including Debezium CDC.
• Own schema governance with Registry and evolution policies.
• Develop stream processing apps using Flink, Kafka Streams, or ksqlDB.
• Design CI/CD and IaC pipelines; monitor cluster health and latency.
• Implement security: encryption, authentication, and private networking.
• Mentor junior engineers and define architecture standards.

Conoscenze

Kafka / Confluent

Java / Python

SQL

Linux

Cloud (AWS/GCP)

CI/CD

Strumenti

Confluent Platform

Confluent Cloud

Kafka

MirrorMaker 2

Replicator

Kafka Connect

ksqlDB

Flink

Descrizione del lavoro

Senior Streaming Data Engineer (Confluent / Apache Kafka)

Experience: 3-8 years

Location: [Bangalore]

Employment type: Full-time

About the Role

We are looking for a Senior Streaming Data Engineer to design, operate, and migrate real-time data platforms on Confluent Platform and Confluent Cloud. You will own critical pipelines that move data between applications, databases, and analytics systems, and lead migrations into and out of Confluent.

Experience with other streaming platforms (Kinesis, Pub/Sub, Event Hubs, Pulsar) is valued, as we want someone who can cross-skill quickly and map new platforms onto Kafka.

Responsibilities

• Design and manage Kafka solutions on Confluent: topic design, partitioning strategy, retention, replication, quotas, and ACLs/RBAC
• Lead end-to-end migrations into and out of Confluent (from Kinesis, Pub/Sub, or self-managed Kafka) using MirrorMaker 2, Replicator, Cluster Linking, or custom bridge consumers, including validation and rollback planning
• Guarantee ordering, idempotency, and consumer offset continuity during cutovers
• Build and review producers, consumers, and Kafka Connect pipelines (Debezium CDC included)
• Own schema governance with Schema Registry, including compatibility policies and evolution
• Develop stream processing applications using Flink, Kafka Streams, or ksqlDB
• Design CI/CD and infrastructure-as-code pipelines for consumers, connector configs, and cluster provisioning
• Monitor cluster health and consumer lag, and tune throughput, latency, and cost
• Implement security: encryption, authentication, and private networking
• Define architecture standards, write runbooks, and mentor junior engineers

Requirements

• 5+ years in data, platform, or distributed systems engineering, including 3+ years with Kafka / Confluent in production
• Deep understanding of Kafka internals and semantics: partitions, consumer groups, rebalancing, offsets, ordering, and exactly-once delivery
• Proven experience with Kafka administration, capacity planning, and troubleshooting production incidents
• Proficiency in Java or Python, strong SQL, and shell scripting
• Strong working knowledge of AWS or GCP (IAM, networking, storage), including VPC, TLS, and private connectivity
• Strong Linux fundamentals: debugging, networking commands, disk and memory behavior
• CI/CD experience (Git with Jenkins, GitHub Actions, or GitLab CI) and monitoring tools (Prometheus, Grafana, Datadog, or CloudWatch)
• Understanding of distributed systems fundamentals: replication, leader election, backpressure, and failure handling

Bonus Skills

Hands-on Kinesis or Pub/Sub, Go, Terraform (Confluent provider), Kubernetes (Confluent for Kubernetes, Strimzi), Snowflake/BigQuery, Confluent certification (Developer or Administrator)