Data Engineer

Technogen; Gurugram District, Bengaluru, Pune District; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

Technogen is seeking an experienced Data Engineer to design, build, and support scalable data pipelines using PySpark and modern Big Data tools. The role focuses on maintaining data processing systems, enabling reliable data ingestion, and optimizing workflows in a cloud environment.

The candidate will work with Apache Iceberg for table management, Airflow for orchestration, AWS EMR for Spark jobs, and Kafka for CDC data streams.

Qualifications

• 6-10 years of experience as a Data Engineer.
• Strong hands-on experience with PySpark and Python.
• Expertise in Apache Iceberg.
• Experience with Apache Airflow.
• Hands-on experience with AWS EMR.
• Strong SQL skills.
• Experience with Kafka and CDC concepts.
• Knowledge of Git and CI/CD.

Responsibilities

• Develop, maintain, and optimize ETL/data processing pipelines using PySpark and Python.
• Build and manage data workflows using Apache Airflow.
• Work with Apache Iceberg for table management, schema evolution, partitioning, bucketing, and time travel.
• Execute and monitor Spark jobs on AWS EMR.
• Support Kafka-based CDC data ingestion pipelines.
• Write optimized SQL queries and implement data transformations using Medallion Architecture (Bronze, Silver, Gold).
• Troubleshoot production issues and provide L3 application support.
• Collaborate with cross-functional teams to ensure data quality, performance, and reliability.
• Follow Git-based version control and CI/CD deployment best practices.

Skills

PySpark

Python

Apache Iceberg

Apache Airflow

AWS EMR

SQL

Kafka

CDC concepts

Git

CI/CD

Spark optimization

Tools

Git

CI/CD

Job description

Job Title

Data Engineer PySpark | Apache Iceberg | AWS EMR | Airflow | Kafka

Job Description

We are hiring experienced Data Engineers to join our team in an L3 Support role. The ideal candidate should have strong expertise in building and supporting scalable data pipelines using modern Big Data and Cloud technologies.

Roles & Responsibilities

• Develop, maintain, and optimize ETL/data processing pipelines using PySpark and Python.
• Build and manage data workflows using Apache Airflow.
• Work with Apache Iceberg for table management, schema evolution, partitioning, bucketing, and time travel.
• Execute and monitor Spark jobs on AWS EMR.
• Support Kafka-based CDC (Change Data Capture) data ingestion pipelines.
• Write optimized SQL queries and implement data transformations using the Medallion Architecture (Bronze, Silver, Gold).
• Troubleshoot production issues and provide L3 application support.
• Collaborate with cross-functional teams to ensure data quality, performance, and reliability.
• Follow Git-based version control and CI/CD deployment best practices.

Mandatory Skills

• 6-10 years of experience as a Data Engineer
• Strong hands-on experience with PySpark and Python
• Expertise in Apache Iceberg
• Experience with Apache Airflow
• Hands-on experience with AWS EMR
• Strong SQL skills
• Experience with Kafka and CDC concepts
• Knowledge of Git and CI/CD

Good to Have

• Experience with Data Lake/Lakehouse architecture
• Knowledge of performance tuning and Spark optimization
• Exposure to Microservices
• Experience supporting production environments

Job Details

• Experience: 6-10 Years
• Employment Type: Permanent
• Work Location: Bangalore, Pune, Chennai, Gurgram & Kolkata (Hyderabad not available)
• Notice Period: Immediate Joiners Preferred