Spark Data Engineer (PySpark) - Cloud ETL & Pipelines

Tata Consultancy Services; Irving (TX); On-site; USD 90,000 - 120,000; Full time

Benefits offered by this job

Discretionary Annual Incentive

Medical Coverage (Health, Dental & Vis

401K Plan

Certification & Training Reimbursement

Job summary

Tata Consultancy Services in Irving, TX seeks a Data Engineer to design, build, and optimize scalable data pipelines using PySpark, Spark SQL, and Hive within cloud environments. The role emphasizes data reliability, speed, and efficiency to support BI and analytics initiatives.

Responsibilities include ETL/ELT development, cloud infrastructure management on AWS/Azure, data warehousing, performance tuning, and collaboration with data scientists and business analysts.

Qualifications

• Proficient in Python and PySpark for complex transformations.
• Strong SQL skills with HiveQL and ANSI SQL.
• Experience with Spark architecture, drivers, executors, and DAGs.

Responsibilities

• Design, build, and maintain scalable ETL/ELT data pipelines using PySpark and Spark SQL.
• Deploy and manage data infrastructure on AWS, Azure, or GCP.
• Optimize data layouts, partitioning, and storage in Hive and cloud lakes.
• Tune Spark jobs and resolve performance bottlenecks using Spark UI.

Skills

Python

PySpark

Spark architecture

SQL

Education

Bachelor of Computer Science

Tools

Apache Spark

Hive

Parquet/ORC/Avro

AWS EMR

Azure Databricks

Airflow

Job description

Tata Consultancy Services in Irving, TX seeks a Data Engineer to design, build, and optimize scalable data pipelines using PySpark, Spark SQL, and Hive within cloud environments. The role emphasizes data reliability, speed, and efficiency to support BI and analytics initiatives.

Responsibilities include ETL/ELT development, cloud infrastructure management on AWS/Azure, data warehousing, performance tuning, and collaboration with data scientists and business analysts.