Data engineer Lead

Indihire Consultants; Hyderabad, Pune District, Bengaluru; Hybrid; INR 1,200,000 - 1,800,000; Full time

Job summary

Indihire Consultants in Hyderabad is seeking a skilled streaming engineer to design and maintain real-time data pipelines using Apache Flink and Apache Kafka. You will build production-grade streaming applications for high-volume, low-latency workloads and implement data transformation, enrichment, and event processing.

The role emphasizes reliable pipelines with robust error handling, proper partitioning and consumer groups, and proactive troubleshooting of failures and performance issues in

Qualifications

• Experience building real-time data pipelines with Flink and Kafka.
• Ability to design streaming applications for high-volume, low-latency workloads.
• Knowledge of event processing, transformations, filtering, enrichment and aggregation.

Responsibilities

• Design, develop, and maintain real-time data pipelines using Apache Flink and Apache Kafka.
• Develop production-grade streaming applications for high-volume and low-latency workloads.
• Implement data transformation, filtering, enrichment, aggregation, and event processing.

Skills

Apache Flink

Apache Kafka

Real-time processing

Streaming apps

Data transformation

Error handling

Job description

Preferred candidate profile

• Design, develop, and maintain real-time data pipelines using Apache Flink and Apache Kafka.
• Develop production-grade streaming applications for high-volume and low-latency workloads.
• Implement data transformation, filtering, enrichment, aggregation, and event processing.
• Build reliable event-processing pipelines with appropriate error handling and recovery mechanisms.
• Consume and publish events across Kafka topics.
• Implement appropriate partitioning, consumer groups, offsets, and delivery mechanisms.
• Troubleshoot streaming pipeline failures and performance issues.

Apache Flink

• Develop and maintain Apache Flink jobs for real-time data processing.
• Implement: Stream transformations