Lead Data Engineer

Indihire Consultants; Hyderabad, Pune District, Bengaluru; Hybrid; INR 3,600,000 - 4,400,000; Full time

Job summary

Indihire Consultants is seeking a Lead Data Engineer in Bengaluru (Hybrid) to own real-time data pipelines and streaming workloads.

You will design, build and operate pipelines with Apache Flink, Apache Kafka, Debezium, ClickHouse, and Airflow, ensuring low latency, high throughput and reliability across source systems to analytics layers.

Qualifications

• 6–8 years of Data Engineering experience.
• Core expertise in Apache Flink and Apache Kafka.
• Proficiency in CDC, Debezium, and ClickHouse.

Responsibilities

• Design and maintain real-time data pipelines with Flink and Kafka.
• Build production-grade streaming applications for high-volume workloads.
• Implement data transformation, enrichment, and event processing.
• Ensure reliability with monitoring, error handling, and recovery.
• Collaborate with APIs, dashboards, and downstream integrations.

Skills

Kafka

Flink

Python

SQL

CDC

Debezium

ClickHouse

Tools

Apache Airflow

Docker

Kubernetes

Job description

Role & responsibilities

Mandatory skills: Kafka, Flink, Python, SQL, CDC, Debezium and Clickhouse

Experience: 3-10 Years

Location: Bangalore/Pune/Hyderabad

For full-time max budget: 40 LPA

Lead Data Engineer Role: Lead Data Engineer Experience: 68 years Location: Bengaluru / Hybrid Employment: Full-time Role Summary We are looking for a Lead Data Engineer with strong hands-on experience in real-time data streaming, event processing, CDC, data integration, and modern data engineering. The Lead Data Engineer will be responsible for designing, developing, and maintaining high performance data pipelines using Apache Flink, Apache Kafka, Debezium, CDC, ClickHouse, and Apache Airflow. This is a hands-on engineering role requiring strong practical experience in building production grade streaming applications, developing data pipelines, troubleshooting distributed systems, and optimizing data processing workloads. The ideal candidate should be comfortable working across the full data pipelinefrom source systems and CDC ingestion through Kafka and Flink processing to analytical storage, APIs, dashboards, and downstream integrations.

Key Responsibilities

Real-Time Data Engineering

• Design, develop, and maintain real-time data pipelines using Apache Flink and Apache Kafka.
• Develop production-grade streaming applications for high-volume and low-latency workloads.
• Implement data transformation, filtering, enrichment, aggregation, and event processing.
• Build reliable event-processing pipelines with appropriate error handling and recovery mechanisms.
• Consume and publish events across Kafka topics.
• Implement appropriate partitioning, consumer groups, offsets, and delivery mechanisms.
• Troubleshoot streaming pipeline failures and performance issues.

Apache Flink

• Develop and maintain Apache Flink jobs for real-time data processing.
• Implement: Stream transformations, Filtering, Mapping, Aggregations, Joins, Windows, Event-time processing, Watermarks, State management.
• Implement Flink checkpointing and recovery mechanisms.
• Optimize Flink jobs for performance, scalability, and resource utilization.
• Monitor Flink jobs for latency, throughput, failures, and resource consumption.
• Troubleshoot state, checkpointing, backpressure, and processing issues.

Kafka

• Develop Kafka-based ingestion and streaming pipelines.
• Create and manage Kafka topics and event streams.
• Work with partitions, offsets, consumer groups, replication, and retention.
• Implement reliable producer and consumer applications.
• Handle message ordering, retries, duplicate events, and replay scenarios.
• Monitor Kafka performance and troubleshoot consumer lag and throughput issues.
• Work with Kafka schemas and serialization formats.

CDC & Debezium

• Build CDC-based ingestion pipelines using Debezium.
• Configure and maintain Debezium connectors.
• Capture source-system inserts, updates, and deletes.
• Publish CDC events into Kafka.
• Handle initial snapshots and incremental CDC processing.
• Manage schema evolution and changes in source systems.
• Implement data reconciliation and consistency checks.
• Troubleshoot CDC failures and source-to-target data issues.

Data Orchestration

• Develop and maintain data workflows using Apache Airflow or equivalent orchestration frameworks.
• Build reusable DAGs for: Data ingestion, CDC workflows, Data validation, Flink job execution, Data transformation, ClickHouse loading, Downstream integrations.
• Implement workflow dependencies, scheduling, retries, backfills, SLAs, and alerting.
• Integrate Airflow workflows with Kafka, Flink, Debezium, ClickHouse, APIs, and cloud services.
• Monitor workflow execution and troubleshoot failures.
• Develop reusable operators, sensors, and workflow components where appropriate.
• Use event-driven triggers where real-time workflows require them.

ClickHouse & Analytical Data

• Integrate streaming data pipelines with ClickHouse.
• Design efficient analytical data models.
• Develop and optimize SQL queries.
• Implement appropriate partitioning, sorting, indexing, and retention strategies.
• Optimize data ingestion and query performance.
• Support downstream analytical use cases, dashboards, and reporting requirements.

Data Quality & Reliability

• Implement data validation and quality checks throughout the pipeline.
• Build reconciliation mechanisms between source and target systems.
• Monitor data freshness, completeness, accuracy, and consistency.
• Implement error handling, retry, replay, and recovery mechanisms.
• Establish logging and observability for critical pipelines.
• Support incident investigation and root-cause analysis.

Integration & APIs

• Integrate streaming and analytical data with APIs, endpoints, dashboards, and downstream applications.
• Develop data interfaces and integration components.
• Work with application teams to define data contracts and integration requirements.
• Support future integrations and additional data consumers.

Engineering Practices

• Follow modern software engineering practices around: Git, Code reviews, Unit testing, Integration testing, CI/CD, Logging, Monitoring, Documentation.
• Develop reusable and maintainable data engineering components.
• Participate in technical design discussions and architecture reviews.
• Mentor other Data Engineers and contribute to engineering standards.

Required Skills & Experience

• 6–8 years of experience in Data Engineering.
• Strong hands-on experience with Apache Flink — mandatory/core requirement.
• Strong hands-on experience with Apache Kafka.
• Hands-on experience with Debezium and CDC.
• Strong programming experience in Java or Scala.
• Python experience is an advantage.
• Strong SQL skills.
• Experience with analytical databases; ClickHouse is highly preferred.
• Hands-on experience with Apache Airflow or another data orchestration framework.
• Strong understanding of distributed systems and real-time data processing.
• Experience developing and supporting production-grade streaming pipelines.
• Strong understanding of Kafka: Topics, Partitions, Offsets, Consumer groups, Replication, Retention.
• Experience with data transformation, enrichment, filtering, aggregation, and event processing.
• Experience troubleshooting performance and reliability issues.
• Familiarity with cloud platforms and containerized environments.

Preferred Skills

• Advanced experience with Apache Flink, including: State management, Checkpoints, Savepoints, Watermarks, Event time, Windows, Backpressure, State backends.
• Experience with Apache Airflow, Dagster, Prefect, or Apache NiFi.
• Experience with Kafka Schema Registry.
• Experience with Avro, Protobuf, or JSON.
• Experience with Kubernetes.
• Experience with AWS / Azure / GCP.
• Experience with CI/CD pipelines.
• Experience with Docker and containerized applications.
• Experience with Terraform or other Infrastructure as Code tools.
• Experience with data observability and monitoring tools.
• Experience building high-volume, low-latency real-time data platforms.
• Experience with REST APIs and system integrations.