Lead Data Engineer

Indus Towers Ltd.; Gurugram District; Presencial; INR 4.200.000 - 6.200.000; Jornada completa

Descripción de la vacante

Indus Towers Ltd. in Gurugram is seeking a Lead Data Engineer to design, build, and optimize scalable data platforms and real-time processing solutions.

You will lead ingestion, transformation, and governance across cloud-native pipelines on AWS, collaborating with architects, data scientists, and business stakeholders. The role requires strong PySpark and SQL expertise, extensive streaming and batch pipeline experience, and hands-on leadership to drive reliable, cost-efficient data ecosystems

Formación

• 7-10 years hands-on data engineering experience.
• Strong Python, PySpark, SQL skills.
• Experience with AWS data services and streaming frameworks.

Responsabilidades

• Design, develop, and maintain scalable batch and real-time data pipelines.
• Build and optimize cloud-native data solutions on AWS.
• Develop and manage streaming data platforms using Kafka, AWS MSK, Kinesis, Spark Streaming or Flink.
• Leverage S3, Redshift, or Aurora for storage and querying of structured data.
• Design and manage Iceberg-based data platforms with schema evolution and ACID transactions.
• Lead data ingestion, transformation, integration, and orchestration across sources.
• Implement idempotent processing, CDC merge strategies, and data reconciliation across pipelines.
• Define and enforce data governance, quality, and metadata standards.
• Collaborate with cross-functional teams to translate business needs into scalable tech solutions.
• Apply performance tuning to improve pipeline efficiency, scalability and cost in AWS.
• Support CI/CD, infrastructure automation, monitoring, and production deployments.
• Troubleshoot complex data engineering challenges and mentor junior engineers.

Conocimientos

Python

PySpark

SQL

Data modeling

ETL/ELT

Data warehousing

Distributed computing

CI/CD

DevOps practices

Streaming concepts

Educación

Bachelor's degree in CS/IT/Engineering

Master's degree

Herramientas

AWS Glue

Amazon S3

Amazon Redshift

Amazon Athena

AWS Lambda

Step Functions

Amazon EMR

Apache Kafka

AWS MSK

AWS Kinesis

Apache Spark Streaming

Apache Flink

Apache Iceberg

Git

CI/CD pipelines

Descripción del empleo

We are seeking a highly experienced and motivated Lead Data Engineer to join our dynamic data platform team in our Gurugram office to drive the design, development, and optimization of scalable data platforms and real-time data processing solutions. The ideal candidate will have extensive experience building modern data pipelines on AWS Cloud, working with streaming technologies, expert-level proficiency in PySpark and SQL, and leading the implementation of enterprise-grade data engineering solutions. As a lead data engineer, you will work closely with data architects, data scientists, analytics teams, and business stakeholders to deliver reliable, scalable, and high-performance data ecosystems that enable data-driven decision-making.

Responsibilities:

• Design, develop, and maintain scalable batch and real-time data pipelines.
• Build and optimize cloud-native data solutions on AWS leveraging modern data engineering best practices.
• Develop and manage streaming data platforms using technologies such as MSK, Firehose, Spark Streaming, or Flink.
• Leverage Amazon S3, Amazon Redshift, or Amazon Aurora for efficient storage and querying of structured data.
• Design and manage Apache Iceberg-based data platforms with support for schema evolution, time travel, partition optimization, and ACID transactions.
• Lead data ingestion, transformation, integration, and orchestration processes across multiple data sources.
• Implement idempotent processing, CDC merge (upsert) strategies, and data reconciliation mechanisms to ensure consistency across batch and streaming pipelines.
• Define and enforce data governance, data quality, and metadata management standards across the data platform.
• Collaborate with cross-functional teams to understand business requirements and translate them into scalable technical solutions.
• Apply performance tuning and optimization techniques to improve pipeline efficiency, scalability, and cost-effectiveness in AWS environments.
• Support CI/CD implementation, infrastructure automation, monitoring, and production deployments.
• Troubleshoot and resolve complex data engineering challenges in production environments.
• Mentor junior and mid-level data engineers and establish engineering standards and best practices.

Requirements:

• Bachelor's or master's degree in computer science, information technology, engineering, or a related field.
• 7-10 years of hands-on experience in data engineering.
• Strong understanding of data engineering concepts, including data modeling, ETL/ELT, data warehousing, and distributed computing.
• Extensive experience working with AWS services such as AWS Glue, S3, Redshift, Athena, Lambda, Step Functions, and Amazon EMR.
• Strong hands-on experience with AWS data services, such as building and managing real-time and streaming data pipelines.
• Hands‑on expertise with streaming frameworks such as Apache Kafka, AWS MSK, AWS Kinesis, Apache Spark Streaming, and Apache Flink.
• Strong programming skills in Python, PySpark, SQL, and experience with data processing frameworks.
• Experience working with large-scale structured and unstructured datasets.
• Strong knowledge of data lake and lakehouse architectures.
• Good to know about building AI‑ready data platforms and understanding multi‑agent frameworks and agentic AI architectures.
• Familiarity with Git, CI/CD pipelines, and DevOps practices.