Big Data Developer - Spark Scala

Tata Consultancy Services; Hyderabad, Chennai District, Bengaluru; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Tata Consultancy Services in Hyderabad is seeking a Big Data Developer to design and deliver scalable data transformation pipelines using Apache Spark with Java/Scala. The role requires hands-on coding, CI/CD integration, and testing to quality standards.

You'll work with Hive, Impala, and HBase to solve large data problems and ensure robust data processing workflows within a modern Hadoop ecosystem.

Qualifications

• Hands-on development with Java/Scala using Maven and Spark frameworks
• Unix Shell scripting and Unix filesystem/HDFS familiarity
• Experience with query languages: Hive SQL, Spark SQL, Impala, HBase

Responsibilities

• Design high quality deliverables adhering to standards and patterns
• Develop and maintain data transformation applications using Apache Spark
• Develop/Integrate code with CI/CD, using Spark in Scala/Java
• Provide solutions to Big data problems using Spark-based transformations, Hive, and Impala
• Create JUnit tests and ensure code coverage meets agreed standards

Skills

Java

Scala

Unix Shell

Spark

Tools

Maven

Apache Spark

Job description

1. Desired Competencies (Technical/Behavioral Competency)

• Hands on development experience in programming languages such as JAVA,SCALA using Maven, Apache Spark Frameworks and Unix Shell scripting
• Should be comfortable with Unix File system as well as HDFS commands
• Should have worked on query languages such as Oracle SQL, Hive SQL, Spark SQL, Impala, HBase DB

Responsibility of / Expectations from the Role

1. Design high quality deliverables adhering to business requirement with defined standards and design principles, patterns

2 Develop and maintain highly scalable, high performance Data transformation applications using Apache Spark framework

3 Develop/Integrate the code adhering to CI/CD, using Spark Framework in Scala/Java

4 Provide solutions to Big data problems dealing with huge volumes of data using Spark based data transformation solutions , Hive, MPP processes like IMPALA.

5 Create Junit tests and ensure code coverage is met as per the agreed standards