Spark Scala Developer

Tata Consultancy Services; India; On-site; INR 900,000 - 1,500,000; Full time

Job summary

Tata Consultancy Services seeks a Data Engineer to design and implement high-quality data transformation solutions using Apache Spark, Scala/Java, and CI/CD practices. You will work on large-scale data with Hive and Impala, ensuring robust code with JUnit tests and scalable architectures.

Join a team that may be geographically distributed and contribute to building performant, standards-driven data pipelines leveraging Spark, HDFS, and related technologies.

Qualifications

• Hands on development experience in JAVA and SCALA.
• Experience with Maven, Apache Spark frameworks and Unix shell scripting.
• Comfortable with Unix filesystem and HDFS commands.
• Experience with Oracle SQL, Hive SQL, Spark SQL, Impala, HBase.
• Strong communication and customer management skills.

Responsibilities

• Design high quality deliverables adhering to standards and design principles.
• Develop and maintain scalable, high performance data transformation applications using Apache Spark.
• Develop/Integrate code adhering to CI/CD, using Spark Framework in Scala/Java.
• Provide solutions to big data problems with Spark-based transformations, Hive, and Impala.
• Create JUnit tests and ensure code coverage meets standards.
• Collaborate with distributed teams and review peers' modules.

Skills

Java

Scala

Apache Spark

Unix Shell

Unix FS

HDFS

Oracle SQL

Hive SQL

Spark SQL

Impala

HBase

JUnit

Communication

Customer Mgmt

Tools

Maven

CI/CD

Job description

Role & responsibilities

• 1Design high quality deliverables adhering to business requirement with defined standards and design principles, patterns
• 2 Develop and maintain highly scalable, high performance Data transformation applications using Apache Spark framework
• 3 Develop/Integrate the code adhering to CI/CD, using Spark Framework in Scala/Java
• 4 Provide solutions to Big data problems dealing with huge volumes of data using Spark based data transformation solutions , Hive, MPP processes like IMPALA.
• 5 Create Junit tests and ensure code coverage is met as per the agreed standards
• 6 Should be able to work with a team who might be geographically distributed. Review the code modules developed by other juniors.

Must have

• Hands on development experience in programming languages such as JAVA,SCALA using Maven, Apache Spark Frameworks and Unix Shell scripting Should be comfortable with Unix File system as well as HDFS commands Should have worked on query languages such as Oracle SQL, Hive SQL, Spark SQL, Impala, HBase DB Should be flexible Should have good communication and customer management skills