Digital : Apache Spark

Tata Consultancy Services; Chennai District; Presencial; INR 1.200.000 - 1.800.000; Jornada completa

Descripción de la vacante

Tata Consultancy Services in Chennai seeks a Big Data Engineer to develop and maintain scalable batch and ETL pipelines using Spark, Hadoop, and Hive. The role focuses on performance tuning, data quality, and efficient resource utilization, working with large datasets across on-premise and cloud data lakes.

The ideal candidate will design and optimize Spark applications, leveraging Spark SQL, RDDs, and Python/Scala to ensure robust data processing workflows.

Formación

• Experience in Big Data technologies with strong expertise in Apache Spark, PySpark/Scala Spark, Hadoop, Hive, and SQL.

Responsabilidades

• Develop and maintain scalable batch and ETL data pipelines using Spark RDDs, DataFrames, Spark SQL, and distributed processing frameworks.
• Design, optimize, and troubleshoot Spark applications with a focus on performance tuning, partitioning, caching, serialization, and resource utilization.
• Work with large-scale structured and unstructured datasets stored in HDFS, Hive, S3, and cloud-based data lakes.
• Develop complex data transformation, aggregation, validation, and reconciliation logic using Spark SQL, SQL, and Scala/Python.

Conocimientos

Big data analytics

ETL design

Performance tuning

Herramientas

Apache Spark

PySpark/Scala Spark

Hadoop

Hive

SQL

Descripción del empleo

Role & responsibilities

• Experience in Big Data technologies with strong expertise in Apache Spark, PySpark/Scala Spark, Hadoop, Hive, and SQL.
• Develop and maintain scalable batch and ETL data pipelines using Spark RDDs, DataFrames, Spark SQL, and distributed processing frameworks.
• Design, optimize, and troubleshoot Spark applications with a focus on performance tuning, partitioning, caching, serialization, and resource utilization.
• Work with large-scale structured and unstructured datasets stored in HDFS, Hive, S3, and cloud-based data lakes.
• Develop complex data transformation, aggregation, validation, and reconciliation logic using Spark SQL, SQL, and Scala/Python.