Hiring Bigdata SME - Chennai/ Bangalore

Tech Mahindra Limited; Chennai, Bengaluru; Hybrid; INR 3,500,000 - 9,000,000; Full time

Job summary

Tech Mahindra Limited is seeking a Data Engineering Hadoop SME with 10+ years of experience to design scalable data solutions. The role emphasizes hands-on development in Hadoop, Spark, Hive, Python/Scala, SQL, and Airflow, and includes guiding development teams and optimizing large-scale data workloads.

You will mentor engineers, collaborate with architects and stakeholders, and contribute to governance, data quality, and architecture documentation.

Qualifications

• 10+ years of Data Engineering/Big Data experience with hands-on development.
• Expert-level knowledge of Hadoop ecosystem and HDFS.
• Strong Spark/PySpark, Hive/HiveQL, Python/Scala, and SQL skills.
• Experience with Airflow or similar orchestration tools and distributed computing.

Responsibilities

• Act as the SME for Hadoop and Big Data engineering solutions.
• Design and develop scalable ETL/data pipelines for large-scale processing.
• Provide technical design guidance using Hadoop, Spark, Hive, and related tech.
• Develop Spark/PySpark/Scala jobs and optimize performance.
• Build and manage Airflow workflows and complex SQL/HiveQL queries.
• Lead performance tuning and troubleshoot production Spark/Hadoop workloads.
• Design ingestion solutions using Kafka, APIs, files, and other sources.
• Contribute to data modelling, data quality, lineage, and governance.
• Conduct code reviews and mentor engineering teams.

Skills

Hadoop ecosystem

Spark / PySpark

Hive / HiveQL

Python / Scala

SQL

Airflow

Performance tuning

Distributed computing

Data modelling

Mentoring/leadership

Tools

Airflow

Kafka

Git

Job description

Job Title: Data Engineering Hadoop SME

Experience: 1015 years
Role: Hadoop / Big Data Subject Matter Expert

Job Summary

We are looking for an experienced Hadoop/Data Engineering SME with strong hands-on expertise in Hadoop, Spark, Hive, Python/Scala, SQL, and Airflow. The role will be responsible for designing scalable data solutions, resolving complex technical challenges, optimizing large-scale data workloads, and providing technical guidance to development teams.

Key Responsibilities

• Act as the SME for Hadoop and Big Data engineering solutions.
• Design and develop highly scalable ETL/data pipelines for large-volume data processing.
• Provide technical design and solution guidance using Hadoop, HDFS, Spark, Hive, and related technologies.
• Develop and optimize complex Spark/PySpark/Scala processing jobs.
• Build and manage workflow orchestration using Airflow.
• Write and optimize complex SQL/HiveQL queries.
• Drive performance tuning and troubleshoot complex production issues involving Spark, Hive, and distributed workloads.
• Design ingestion solutions for structured and unstructured data using Kafka, APIs, files, and other sources.
• Provide expertise in data modelling, data quality, lineage, and governance.
• Conduct technical reviews, code reviews, and enforce engineering best practices.
• Mentor developers and provide technical guidance to engineering teams.
• Collaborate with architects, business stakeholders, and upstream/downstream teams on solution design.
• Create and maintain technical design and architecture documentation.

Mandatory Skills

• 10+ years of Data Engineering/Big Data experience, with strong hands-on development expertise.
• Expert-level knowledge of Hadoop ecosystem and HDFS.
• Strong hands-on experience with Apache Spark / PySpark.
• Strong knowledge of Hive / HiveQL.
• Strong programming skills in Python and/or Scala.
• Advanced SQL and strong data modelling fundamentals.
• Hands-on experience with Airflow or similar orchestration platforms.
• Strong experience in performance tuning and troubleshooting Spark/Hadoop workloads.
• Good understanding of distributed computing and large-scale data processing.
• Linux and Shell scripting knowledge.
• Experience with Git and Agile development practices.