Scala Lead

Tata Consultancy Services; Mumbai; On-site; INR 2,250,000 - 2,750,000; Full time

Job summary

Tata Consultancy Services in Mumbai seeks a seasoned Big Data Engineer to design and implement Spark-based data pipelines. You will transform and aggregate large datasets using Scala and Spark, ensuring reliable batch and streaming processing. 

The role requires hands-on expertise with Hadoop components, Hive SQL, and scripting. Collaborate in an Agile environment and document mappings and transformations thoroughly.

Qualifications

• 5+ years in Spark Scala development.
• Experience with Hadoop ecosystem: HDFS, Spark, Hive, Parquet, YARN, MapReduce, Sqoop.
• Proficient in writing and optimizing Spark Jobs and Spark SQL for batch and streaming processing.
• Proficient in Hive SQL, UDFs, tables, joins, and views.
• Experience in debugging Spark code; basic UNIX shell scripting.
• Experience with Autosys and Gradle.

Responsibilities

• Create Scala/Spark jobs for data transformation and aggregation.
• Produce unit tests for Spark transformations and helper methods.
• Write Scaladoc-style documentation with all code.

Skills

Spark Scala

Batch processing

Streaming data

Spark SQL

UNIX shell

Tools

Hadoop

HDFS

Hive

Parquet

YARN

MapReduce

Sqoop

Autosys

Gradle

Job description

Desired Competencies (Technical/Behavioral Competency)

Must-Have**

(Ideally should not be more than 3-5)

• Minimum 5+ years of experience in development of Spark Scala
• Experience in designing and development of solutions for Big Data using Hadoop ecosystem technologies such as with Hadoop Bigdata components like HDFS, Spark, Hive Parquet File format, YARN, MapReduce, Sqoop
• Good Experience in writing and optimizing Spark Jobs, Spark SQL etc. Should have worked on both batch and streaming data processing.
• Experience in writing and optimizing complex Hive and SQL queries to process huge data. good with UDFs, tables, joins, Views etc
• Experience in debugging the Spark code
• Working knowledge of basic UNIX commands and shell script
• Experience of Autosys, Gradle

Good-to-Have

• Good analytical and debugging skills
• Ability to coordinate with SMEs, stakeholders, manage timelines, escalation & provide on time status
• Write clear and precise documentation / specification
• Work in an agile environment
• Create documentation and document all developed mappings

SN

Responsibility of / Expectations from the Role

1

Create Scala/Spark jobs for data transformation and aggregation

2

Produce unit tests for Spark transformations and helper methods

3

Write Scaladoc-style documentation with all code