Lead Engineer

Tech Mahindra; Bengaluru; On-site; INR 3,500,000 - 6,500,000; Full time

Job summary

Tech Mahindra in Bengaluru is seeking a Data Engineering Lead to head a team building scalable ingestion and transformation pipelines on Databricks and AWS, aligning with architecture and delivery timelines. You will lead a pod delivering Bronze/Silver/Gold pipelines across batch and streaming patterns, drive design reviews for ingestion and streaming jobs, and manage sprint planning and delivery.

Collaborate with Data Architect, ensure governance with Unity Catalog, data quality checks, and

Qualifications

• 7–10 years of data engineering experience, with at least 2 years in team leadership.
• Hands-on expertise with Databricks, Spark, Delta Lake, SQL, and Python/PySpark.
• Strong experience with AWS data services (Glue, S3, DMS, DataSync) and streaming (Kinesis/Kafka).
• Solid knowledge of medallion architecture, CDC ingestion, and orchestration tools.
• Proven ability to deliver on timelines and mentor engineers.

Responsibilities

• Lead a data engineering pod delivering Bronze/Silver/Gold pipelines.
• Own sprint planning, task allocation, and delivery tracking.
• Oversee ingestion pipelines and streaming jobs with AWS services.
• Ensure governance and data quality with Airflow/MWAA.
• Mentor junior engineers and collaborate with Data Architect.
• Communicate progress and blockers to clients/stakeholders.

Skills

Databricks / Spark

Delta Lake

SQL

Python / PySpark

AWS data services

Kinesis / Kafka

Medallion architecture

Airflow / MWAA

Leadership / mentoring

Tools

Databricks

AWS Glue

DMS

DataSync

Kinesis

Kafka

Airflow / MWAA

Job description

Role Summary

We are looking for a Data Engineering Lead to lead a team of data engineers in building and delivering scalable ingestion and transformation pipelines on Databricks + AWS, ensuring engineering execution aligns with the platform's architecture and delivery timelines.

Key Responsibilities

• Lead a team of Data Engineers delivering Bronze/Silver/Gold pipelines across batch, near-real-time, streaming, and CDC patterns
• Drive hands-on implementation and code/design reviews for ingestion pipelines (AWS Glue, DMS, DataSync) and streaming jobs (Kinesis, Kafka, or Databricks Structured Streaming)
• Own sprint planning, task allocation, and delivery tracking for the engineering pod
• Ensure adherence to Unity Catalog governance standards, data quality checks, and orchestration best practices (Airflow/MWAA)
• Troubleshoot complex pipeline/performance issues and mentor junior engineers
• Collaborate closely with the Data Architect to translate architecture into engineering execution
• Support client/stakeholder updates on engineering progress and technical blockers

Must-Have Skills

• 7-10 years in data engineering, including 2+ years leading a team/pod
• Strong hands-on Databricks/Spark, Delta Lake, SQL, and Python/PySpark expertise
• Strong AWS data services experience: Glue, S3, DMS; working knowledge of Kinesis/MSK or Kafka
• Solid understanding of medallion architecture, CDC ingestion, and orchestration tools
• Proven ability to manage delivery timelines and mentor a team

Good-to-Have: Exposure to vector databases, RAG pipeline data prep, or agentic/AI tooling (Bedrock, Mosaic AI).