Senior Data Engineer: Spark & Cloud Pipelines

Tata Consultancy Services; Irving (TX); On-site; USD 90,000 - 110,000; Full time

Benefits offered by this job

Discretionary Annual Incentive.

Comprehensive Medical Coverage

401K Plan

Certification & Training Reimbursement

Job summary

Tata Consultancy Services is seeking a Data Engineer to design, build, and optimize scalable data pipelines using Spark, PySpark, and Hive in a cloud environment, delivering reliable and fast data for analytics.

You will manage ETL/ELT workflows, tune Spark jobs, oversee cloud infrastructure across AWS, Azure, or GCP, and collaborate with data scientists and analysts to translate business needs into robust data solutions.

Qualifications

• Proficient in Apache Spark architecture and DAGs.
• Strong Python and PySpark programming skills.
• Expert in HiveQL and ANSI SQL with partitioning.
• Experience with Parquet, ORC, Avro formats.
• Cloud data infrastructure on AWS/Azure/GCP.

Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines using PySpark and Spark SQL.
• Manage cloud data infrastructure on AWS, Azure, or GCP.
• Optimize data warehousing and storage in Hive and data lakes.
• Tune Spark jobs and address data skew and memory usage.
• Ingest high-volume diverse datasets from relational and flat files.
• Coordinate data workflows using Airflow or platform schedulers.
• Collaborate with data scientists and analysts to translate requirements into data solutions.

Skills

Spark architecture

Python / PySpark

HiveQL / ANSI SQL

Parquet/ORC/Avro

Cloud data infrastructure

Dimensional modeling

Education

Bachelor of Computer Science

Tools

Apache Airflow

AWS EMR

Azure Databricks

Git

Jenkins

Ansible

Job description

Tata Consultancy Services is seeking a Data Engineer to design, build, and optimize scalable data pipelines using Spark, PySpark, and Hive in a cloud environment, delivering reliable and fast data for analytics.

You will manage ETL/ELT workflows, tune Spark jobs, oversee cloud infrastructure across AWS, Azure, or GCP, and collaborate with data scientists and analysts to translate business needs into robust data solutions.