Senior Data Engineer Kafka and Streaming Data Processing

Nihilent Ltd.; Pune District; On-site; INR 3,000,000 - 4,500,000; Full time

Job summary

Nihilent Ltd. is seeking a Senior Data Engineer to design and scale real-time data pipelines using Apache Kafka and Debezium on AWS and an open-source stack.

You will work across the AWS data ecosystem to implement streaming architectures and ensure reliable, high-quality data processing. The role requires 10–12 years of data engineering experience, deep Kafka and CDC expertise, and strong PySpark, SQL, Airflow, Iceberg, and Redshift skills.

Qualifications

• 10-12 years of data engineering experience with hands-on Kafka, CDC, and AWS skills (must-have).
• Experience with Debezium / CDC tools.
• Proficiency in PySpark, SQL, AWS Airflow, Glue, Iceberg and Redshift.

Responsibilities

• Design & manage Kafka-based streaming pipelines (topics, partitions, consumer groups).
• Design & manage Spark-based batch pipelines using Spark framework.
• Configure and maintain Debezium connectors for real-time CDC.
• Build scalable ETL/ELT workflows using PySpark and AWS Airflow.
• Strong experience with Iceberg tables and AWS open source stack.
• Manage and optimize AWS Redshift for analytics and reporting.
• Ensure pipeline reliability, data quality, and performance at scale.
• Collaborate with cross-functional teams on data requirements.

Skills

Kafka

Debezium

PySpark

SQL

AWS

Streaming data

Education

B.E./B.Tech in CS or related

Tools

Apache Airflow

Iceberg

Redshift

S3

Job description

Senior Data Engineer Kafka & Streaming Data Processing

Location: Pune

Experience

10-12 Years

About the Role:

We're hiring a Senior Data Engineer to build and scale real-time data processing pipelines using Apache Kafka and Debezium on AWS and open source stack like Apache Iceberg, Open metadata, Atlas, Apache Airflow. If you're passionate about streaming architectures and enjoy working across the AWS data ecosystem and open source stack, this role is for you.

Responsibilities:

• Design & manage Kafka-based streaming pipelines (topics, partitioning, consumer groups), Flink
• Design & manage Spark based Batch pipelines using Spark based framework
• Configure and maintain Debezium connectors for real-time CDC
• Build scalable ETL/ELT workflows using PySpark and AWS Airflow
• Strong experience in utilizing Iceberg tables and AWS open source system.
• Manage and optimize AWS Redshift for analytics & reporting
• Ensure pipeline reliability, data quality, and performance at scale
• Collaborate with cross-functional teams on data requirements

What We're Looking For:

• 10-12 years of Data Engineering experience with strong hands-on Kafka, CDC, AWS skills (must-have)
• Experience with Debezium / CDC tools
• Proficiency in PySpark, SQL, AWS Airflow, Glue, Iceberg and Redshift
• Working knowledge of AWS services (S3, IAM, Lambda)
• Strong SQL and data modeling fundamentals

Nice to Have:

• Experience in the Finance/Banking domain