Pyspark Spark Python Developer - Chennai/ Bangalore

Tech Mahindra; Chennai District; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

Tech Mahindra seeks a Hadoop Developer with 8 years of experience to design, build, and optimize scalable data pipelines on Hadoop ecosystem technologies. The role covers batch/stream processing, data integration, and close collaboration with business teams.

The candidate will design ETL/data pipelines, optimize large-scale Spark/Hive jobs, and implement Airflow workflows while ensuring data quality and governance.

Qualifications

• 8 years of hands-on experience in Hadoop/big data development.
• Strong knowledge of HDFS, Hive, Spark.
• Proficiency in SQL/HiveQL for data transformation and reporting.

Responsibilities

• Design and develop robust ETL/data pipelines using Hadoop ecosystem tools.
• Build and optimize large-scale data processing jobs using Spark and/or Hive.
• Develop workflows using Airflow.
• Write complex SQL/HiveQL for data transformation and reporting needs.
• Ingest structured and unstructured data from multiple sources (Kafka, Sqoop, APIs, files).
• Ensure data quality, lineage, and governance standards are followed.
• Monitor, troubleshoot, and tune jobs for performance and scalability.
• Create technical documentation and follow coding best practices.

Skills

SQL

Python

Spark

pyspark

Airflow

Education

Bachelor's degree or higher

Tools

Hadoop

Hive

Spark

Airflow

Kafka

Sqoop

Job description

A Bachelor’s or Higher Degree is the minimum entry required for the position

Image

Job Description

• Skill Set : SQL,Python,Spark,pyspark,Airflow

Seeking a Hadoop Developer with 8 years of experience in big data engineering to design, build, and optimize scalable data pipelines on Hadoop ecosystem technologies. The role involves batch/stream processing, data integration, performance tuning, and close collaboration with business teams.

Key Responsibilities

• Design and develop robust ETL/data pipelines using Hadoop ecosystem tools.
• Build and optimize large-scale data processing jobs using Spark and/or Hive.
• Develop workflows using Airflow.
• Write complex SQL/HiveQL for data transformation and reporting needs.
• Ingest structured and unstructured data from multiple sources (Kafka, Sqoop, APIs, files).
• Ensure data quality, lineage, and governance standards are followed.
• Monitor, troubleshoot, and tune jobs for performance and scalability.
• Create technical documentation and follow coding best practices.

Required Skills

• 8 years of hands-on experience in Hadoop/big data development.
• Strong knowledge of HDFS, Hive, Spark,
• String knowledge in Python/Scala
• Strong SQL and data modelling fundamentals.
• Experience with workflow orchestration tools (Airflow).
• Good understanding of Linux, shell scripting, and distributed systems.
• Familiarity with version control (Git) and Agile delivery practices.