Senior Data Engineer(Hadoop + PySpark)

Alike Thoughts; Hyderabad, Chennai District; Hybrid; INR 2,500,000 - 4,200,000; Full time

Job summary

Alike Thoughts is seeking a Senior Data Engineer with 8–12 years of experience to design, develop, and optimize scalable Hadoop and PySpark-based data pipelines for enterprise analytics.

You will implement Spark SQL and PySpark jobs, automate data workflows with Python, monitor clusters, mentor teammates, conduct code reviews, and collaborate with analysts to deliver actionable insights, while upholding security and governance standards.

Qualifications

• 8–12 years of relevant experience in big data engineering.
• Proven expertise in Hadoop + PySpark with large-scale data processing.
• Strong problem-solving and performance tuning skills.
• Ability to mentor and lead technical teams.

Responsibilities

• Design, develop, and deploy scalable big data solutions using Hadoop and Spark ecosystems.
• Implement and optimize PySpark and Spark SQL jobs for performance and resource utilization.
• Develop Python scripts to automate data processing and integration within Hadoop ecosystem.
• Integrate data ingestion and transformation tools such as HDFS, Hive, Sqoop, Oozie, and Kafka.
• Monitor and maintain Hadoop clusters for stability, capacity planning, and performance tuning.
• Collaborate with data engineers, analysts, and stakeholders to deliver actionable insights.
• Conduct code reviews, provide technical guidance, and mentor junior team members.
• Document technical specifications, processes, and best practices.
• Ensure compliance with security and governance standards in big data environments.

Skills

Hadoop ecosystem

PySpark

Spark SQL

Python scripting

Data pipeline design

Tools

HDFS

MapReduce

Hive

Sqoop

Oozie

Kafka

Job description

Senior Data Engineer Hadoop + PySpark

Location: Bangalore /Pune / Chennai / Mumbai /Coimbatore Grade: P4 (812 years)

Job Summary

We are seeking a Senior Data Engineer with 812 years of experience in Hadoop and PySpark to design, develop, and optimize scalable big data solutions. The role involves building highperformance data pipelines, ensuring data quality, and collaborating with crossfunctional teams to deliver enterprisegrade analytics solutions.

Key Responsibilities

• Design, develop, and deploy scalable big data solutions using Hadoop and Spark ecosystems
• Implement and optimize PySpark and Spark SQL jobs for performance and resource utilization
• Develop Python scripts to automate data processing and integration within Hadoop ecosystem
• Integrate data ingestion and transformation tools such as HDFS, Hive, Sqoop, Oozie, and Kafka
• Monitor and maintain Hadoop clusters for stability, capacity planning, and performance tuning
• Collaborate with data engineers, analysts, and stakeholders to deliver actionable insights
• Conduct code reviews, provide technical guidance, and mentor junior team members
• Document technical specifications, processes, and best practices
• Ensure compliance with security and governance standards in big data environments

Mandatory Skills

• Hadoop ecosystem (HDFS, MapReduce, Hive, Sqoop, Oozie, Kafka)
• PySpark and Spark SQL
• Strong Python scripting for data workflows
• Data pipeline design, optimization, and troubleshooting

Candidate Profile

• 8–12 years of relevant experience in big data engineering
• Proven expertise in Hadoop + PySpark with largescale data processing
• Strong problemsolving and performance tuning skills
• Ability to mentor and lead technical teams