Senior PySpark Data Engineer — Scalable Cloud Pipelines

Tata Consultancy Services Limited; New York (NY); Presencial; USD 120.000 - 180.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Discretionary Annual Incentive

Medical Coverage

Parental Leaves

Insurance Options

Commuter Benefits

Certification Training Reimbursement

Vacation & Holidays

401K Plan

Descripción de la vacante

Tata Consultancy Services Limited is seeking a Data Engineer to design, build, and optimize scalable data pipelines in a cloud environment using Apache Spark, PySpark, Hive, and SQL. The role emphasizes data reliability, performance tuning, and collaboration with data science teams.

Responsibilities include developing ETL/ELT processes, managing cloud data infrastructure, optimizing data storage, and implementing automated workflows with Airflow or platform schedulers.

Formación

• Experience building scalable ETL/ELT pipelines with PySpark.
• Strong SQL skills (HiveQL/ANSI SQL) and data modeling fundamentals.
• Experience with cloud data platforms (AWS/Azure/GCP) and data lake architectures.
• Familiarity with Spark performance tuning and memory management.

Responsabilidades

• Design, build, and maintain scalable ETL/ELT pipelines using PySpark.
• Manage cloud data infrastructure on AWS, Azure, or GCP.
• Optimize data layouts, partitioning, and indexing in Hive and data lakes.
• Tune Spark jobs and analyze performance using Spark UI.
• Ingest high-volume structured and unstructured data into the data ecosystem.
• Collaborate with data scientists and analysts to translate requirements into data solutions.
• Implement and manage automated data workflows with Airflow or schedulers.

Conocimientos

Apache Spark

PySpark

Hive

Python

ANSI SQL

Cloud data platforms

Herramientas

Airflow

Databricks

AWS EMR

Azure Databricks

Descripción del empleo

Tata Consultancy Services Limited is seeking a Data Engineer to design, build, and optimize scalable data pipelines in a cloud environment using Apache Spark, PySpark, Hive, and SQL. The role emphasizes data reliability, performance tuning, and collaboration with data science teams.

Responsibilities include developing ETL/ELT processes, managing cloud data infrastructure, optimizing data storage, and implementing automated workflows with Airflow or platform schedulers.