Senior PySpark Engineer | Scalable ETL Pipelines & Cloud

Tata Consultancy Services; Irving (TX); On-site; USD 90,000 - 150,000; Full time

Benefits offered by this job

Discretionary Annual Incentive

Comprehensive Medical Coverage

Parental Leaves

Auto & Home Insurance Options

Certification & Training Reimbursement

Vacation & Holidays

401K Plan

Job summary

Tata Consultancy Services is seeking a Data Engineer to design, build, and optimize scalable data pipelines in a cloud environment. You will work with PySpark, Spark SQL, and HiveQL to ensure data reliability and performance for BI and analytics initiatives.

Ideal candidates will have hands-on experience with AWS/Azure/GCP, data warehousing fundamentals, and a strong focus on performance tuning and data quality.

Qualifications

• Strong experience building scalable data pipelines and ETL/ELT processes.
• Experience with Spark, PySpark, HiveQL, and SQL in cloud environments.
• Knowledge of data warehousing concepts, data partitioning, and schema design.

Responsibilities

• Design, build, and optimize scalable ETL/ELT pipelines using PySpark and Spark SQL.
• Manage and scale cloud data infrastructure (AWS, Azure, GCP).
• Optimize data storage with Hive, Parquet/ORC/Avro formats.
• Tune performance and resolve bottlenecks in Spark jobs.

Skills

Apache Spark

Python/PySpark

HiveQL/SQL

Data Modeling

Cloud Platforms

Spark Tuning

Data Pipelines

Education

Bachelor's degree in Computer Science or related field

Tools

Airflow

ETL/ELT Tools

Parquet/ORC/Avro

Spark UI

Job description

Tata Consultancy Services is seeking a Data Engineer to design, build, and optimize scalable data pipelines in a cloud environment. You will work with PySpark, Spark SQL, and HiveQL to ensure data reliability and performance for BI and analytics initiatives.

Ideal candidates will have hands-on experience with AWS/Azure/GCP, data warehousing fundamentals, and a strong focus on performance tuning and data quality.