Apache Flink Transformation Engineer

Genpact; Dadri, Hyderabad, Bengaluru; On-site; INR 900,000 - 1,500,000; Full time

Job summary

Genpact is seeking a skilled data engineering professional to design, develop, and maintain real-time streaming applications. You will work with Apache Flink and Kafka ecosystems to implement event-driven data pipelines and low-latency processing at scale.

The role involves collaborating with solution architects and business stakeholders to define transformation rules, optimize performance, and ensure reliable streaming operations across enterprise platforms.

Qualifications

• Experience in real-time streaming data engineering.
• Hands-on with Apache Flink for enterprise-scale processing.
• Strong knowledge of Kafka ecosystem: Kafka Streams, KSQLDB, Connect, SMTs.
• Experience designing event-driven architectures and real-time data integration pipelines.
• Understanding of event-time processing, watermarks, windowing, and stateful streaming.

Responsibilities

• Design, develop, and maintain real-time data processing apps using Flink.
• Develop in-flight transformation logic with Flink, Kafka Streams, KSQLDB, SMTs.
• Build scalable mapping frameworks for transforming source systems to enterprise models.
• Implement complex event processing, enrichment, filtering, aggregation, and windowing.
• Optimize streaming pipelines for low latency and high throughput.
• Handle late-arriving events with event-time processing and state management.
• Integrate streaming apps with Kafka, Schema Registry, APIs, and analytics platforms.
• Implement error handling, DLQ, retries, and monitoring for streaming apps.
• Collaborate with architects, API teams, and stakeholders on transformation rules.
• Develop automated testing and CI/CD for streaming apps.
• Monitor production jobs and troubleshoot performance and reliability.

Skills

Apache Flink

Apache Kafka

KSQLDB

Java/Scala

Python

SQL

Cloud platforms

Tools

Schema Registry

Avro/Protobuf/JSON

Docker/Kubernetes

Job description

Key Responsibilities

• Design, develop, and maintain high-performance real-time data processing applications using Apache Flink.
• Develop in-flight transformation logic leveraging Apache Flink, Kafka Streams, KSQLDB, and Kafka Single Message Transforms (SMTs).
• Build and maintain scalable mapping frameworks for transforming source system data into harmonized enterprise data models.
• Implement complex event processing, enrichment, filtering, aggregation, and windowing logic for streaming workloads.
• Optimize streaming pipeline performance, ensuring low latency, high throughput, and efficient resource utilization.
• Design solutions to handle late-arriving and out-of-order events using event-time processing, watermarks, checkpointing, and state management.
• Integrate streaming applications with Kafka, Schema Registry, APIs, and downstream analytical platforms.
• Implement robust error handling, retry mechanisms, dead-letter queues (DLQ), and monitoring for streaming applications.
• Collaborate with Solution Architects, Data Architects, API teams, and business stakeholders to define transformation rules and streaming integration patterns.
• Develop automated testing frameworks and CI/CD pipelines for streaming applications.
• Monitor production streaming jobs and troubleshoot performance, scalability, and reliability issues.

Required Skills & Experience

• 610+ years of experience in Data Engineering with strong expertise in real-time streaming solutions.
• Hands-on experience with Apache Flink for enterprise-scale stream processing.
• Strong knowledge of Apache Kafka, Kafka Streams, KSQLDB, Kafka Connect, and Single Message Transforms (SMTs).
• Experience designing event-driven data architectures and real-time data integration pipelines.
• Strong understanding of event-time processing, watermarks, windowing, checkpointing, stateful stream processing, and fault tolerance.
• Experience developing data transformation and harmonization frameworks for enterprise data platforms.
• Proficiency in Java and/or Scala; Python knowledge is desirable.
• Experience with Avro, Protobuf, JSON, Schema Registry, and serialization techniques.
• Strong SQL skills and experience working with large-scale structured and semi-structured datasets.
• Experience with cloud platforms such as Azure, AWS, or GCP is preferred.
• Familiarity with CI/CD, Git, Docker, Kubernetes, and monitoring tools is an advantage.

Preferred Qualifications

• Experience with Confluent Platform and enterprise Kafka deployments.
• Exposure to modern data lakehouse platforms such as Databricks, Snowflake, or Delta Lake.
• Knowledge of CDC patterns, API integration, and enterprise integration architectures.
• Experience working in Agile/Scrum delivery environments.
• Excellent analytical, problem-solving, and stakeholder communication skills.