Data Engineer: Scalable Spark Pipelines & Cloud Infra

Tata Consultancy Services Limited; Irving (TX); On-site; USD 70,000 - 80,000; Full time

Job summary

Tata Consultancy Services Limited in Irving, Texas seeks a Data Engineer to design, build and optimize scalable data pipelines using PySpark, Spark SQL and Hive in a cloud environment. You will ensure data reliability, speed and accessibility to support BI and advanced analytics.

Responsibilities include ETL/ELT development, cloud infrastructure management on AWS/Azure/GCP, data warehousing tasks, performance tuning of Spark jobs, data ingestion from diverse sources, and collaboration with data

Qualifications

• Big Data Frameworks Expertise: Proficiency in Apache Spark architecture, including drivers, executors, and DAGs.

Responsibilities

• Design, build, and maintain scalable ETL/ELT data pipelines using PySpark and Spark SQL for complex transformations.

Skills

Python programming

SQL

Spark architecture

Data modeling

ETL/ELT design

Big data concepts

Education

Bachelor's degree in Computer Science

Tools

Apache Spark

PySpark

Hive

Spark SQL

Parquet

ORC

Avro

AWS EMR

Azure Databricks

Databricks

Synapse

Airflow

Job description

Tata Consultancy Services Limited in Irving, Texas seeks a Data Engineer to design, build and optimize scalable data pipelines using PySpark, Spark SQL and Hive in a cloud environment. You will ensure data reliability, speed and accessibility to support BI and advanced analytics.

Responsibilities include ETL/ELT development, cloud infrastructure management on AWS/Azure/GCP, data warehousing tasks, performance tuning of Spark jobs, data ingestion from diverse sources, and collaboration with data