Big Data & Hadoop Developer

Tata Consultancy Services Limited; Mumbai; On-site; INR 1,200,000 - 2,000,000; Full time

Job summary

Tata Consultancy Services Limited in Mumbai is seeking experienced Big Data & Hadoop Developers to design, develop, and optimize scalable data-processing solutions using Hadoop ecosystem tools such as Spark, Hive, and SQL.

Ideal candidates will have 5–12 years of hands-on experience delivering large-scale data pipelines, strong Unix/Linux scripting skills, and proficiency with Spark, HiveQL, and data ingestion using Sqoop/Flume.

Qualifications

• Strong hands-on Hadoop/HDFS development experience.
• Experience with Spark, Hive, and large-scale batch processing.
• Unix/Linux scripting proficiency.
• Designing and maintaining large-scale data-processing pipelines.
• SDLC and Agile development practices.

Responsibilities

• Design, develop, test, and maintain scalable Big Data apps.
• Develop distributed data-processing solutions using Hadoop, HDFS, Spark, and Hive.
• Build and maintain reliable batch and large-scale data-processing pipelines.
• Develop and optimize Spark applications, Spark SQL, Hive queries, and UDFs.
• Performance tuning of Spark and Hive workloads.
• Process and transform high-volume datasets efficiently.
• Work with Hadoop ecosystem components for data ingestion, processing, and workflow orchestration.
• Identify performance bottlenecks and implement optimization techniques.
• Participate in requirements analysis, technical design, development, and deployment.
• Follow SDLC and Agile development practices.

Skills

Big Data concepts

Hadoop & HDFS

YARN & MapReduce

Apache Spark

Spark SQL

Hive & HiveQL

Unix/Linux scripting

ETL pipelines

Debugging & performance tuning

SDLC & Agile

Data processing pipelines

SQL

Tools

Apache Sqoop

Apache Flume

Apache Kafka

Autosys

Java

Python

Job description

Role: Developer

Primary Skill: BigData and Hadoop Ecosystems

Experience: 5–12 Years

Locations: Mumbai | Chennai | Hyderabad | Bangalore | Indore | Ahmedabad

Job Description

We are looking for experienced Big Data & Hadoop Developers with strong hands-on expertise in designing, developing, optimizing, and supporting scalable data-processing solutions using Big Data technologies and the Hadoop ecosystem.

The ideal candidate should have strong development experience with Hadoop/HDFS, Apache Spark, Hive, SQL, Unix/Linux scripting, and large-scale distributed data-processing applications.

Must-Have Skills

• Strong understanding of Big Data and distributed computing concepts
• Strong hands-on experience with Hadoop and HDFS
• Good understanding of YARN and MapReduce
• Strong hands-on development experience with Apache Spark
• Experience with Spark SQL, DataFrames, and large-scale batch processing
• Strong hands-on experience with Apache Hive and HiveQL/SQL
• Experience developing and optimizing Hive queries and UDFs
• Strong experience with Unix/Linux scripting
• Experience designing, developing, and supporting large-scale data-processing pipelines
• Strong debugging, troubleshooting, and performance-tuning skills, particularly across Spark and Hive
• Experience working with high-volume structured and semi-structured datasets
• Good understanding of SDLC and software development practices

Big Data & Hadoop Ecosystem Skills

Experience with the following technologies will be advantageous:

• Apache Sqoop for data ingestion and integration
• Apache Flume for data collection and ingestion
• Apache Kafka for messaging and streaming use cases
• Autosys or similar enterprise job-scheduling tools
• Java and/or Python for Big Data application development

Key Responsibilities

• Design, develop, test, and maintain scalable Big Data applications and data-processing solutions.
• Develop distributed data-processing solutions using Hadoop, HDFS, Spark, and Hive.
• Build and maintain reliable batch and large-scale data-processing pipelines.
• Develop and optimize Spark applications, Spark SQL, Hive queries, and UDFs.
• Performance tuning of Spark and Hive workloads.
• Process and transform high-volume datasets efficiently.
• Work with Hadoop ecosystem components for data ingestion, processing, and workflow orchestration.
• Identify performance bottlenecks and implement appropriate optimization techniques.
• Participate in requirements analysis, technical design, development, testing, deployment, and production support.
• Follow established SDLC and Agile development practices.
• Collaborate effectively with technical and business stakeholders.
• Work independently and take ownership of assigned development deliverables.

Good-to-Have Skills

• Apache Kafka and real-time/streaming data-processing concepts
• Spark Structured Streaming
• Sqoop, Flume, and Oozie
• Autosys or equivalent scheduling tools
• Exposure to cloud-based Big Data environments
• Experience supporting production Big Data applications