Support Executive - Big Data & Hadoop Ecosystems (PySpark Developer)

Tata Consultancy Services; Mumbai, Chennai District, Bengaluru; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Tata Consultancy Services in Mumbai is seeking a seasoned Big Data Engineer to work with Hadoop, Spark, Hive, and PySpark. You will handle large-scale data processing, develop Spark jobs, and optimize data workflows across platforms such as HDFS, Sqoop, Flume, and Oozie/Autosys.

The role requires hands-on expertise with Spark and Python, strong debugging skills, and the ability to work independently with client partners.

Qualifications

• Good understanding of Hadoop concepts including file system and Map Reduce.
• Hands-on experience with Spark framework, Unix scripting, Hive queries, wriring UDF in Hive. Theortical knowledge and POC alone will not suffice.
• Good knowledge in Software Development Life Cycle and Project Development Lifecycle.
• Associate should be able to work independently and should have strong debugging skill in both Hive and Spark. Associate should have experience developing large scale systems, experience debugging and performance tuning, excellent software design skills, communication skills and ability to work with client partners. Hands on in Apache Spark with Python or any other language (Preferred is Python -PySpark)

Skills

Hadoop concepts

Spark framework

Hive queries

PySpark

Unix scripting

POC development

Large-scale systems

Performance tuning

Software development lifecycle

Client communication

Tools

Autosys

Oozie

Sqoop

Flume

Kafka

Job description

Required Technical Skill Set

Big Data PySpark, HDFS, Hive, Spark, Sqoop, Flume, Oozie, Unix Script, Autosys

Desired Experience Range: 5-12

Desired Competencies (Technical/Behavioral Competency)

Must-Have

• Good understanding of Hadoop concepts including file system and Map Reduce.
• Hands on experience on Spark framework, Unix scripting, Hive queries, wriring UDF in Hive. Theortical knowledge and POC alone will not suffice.
• Good knowledge in Software Development Life Cycle and Project Development Lifecycle.
• Associate should be able to work independently and should have strong debugging skill in both Hive and Spark. Associate should have experience developing large scale systems, experience debugging and performance tuning, excellent software design skills, communication skills and ability to work with client partners. Hands on in Apache Spark with Python or any other language (Preferred is Python -PySpark)

Good-to-Have

• Experience in Banking and Finance Domain.
• Experience in Agile Methodology
• Knowledge in job scheduling tools like Autosys.
• Knowledge in Kafka.