Senior PySpark Data Engineer

Tata Consultancy Services; Irving (TX); On-site; USD 100,000 - 130,000; Full time

Job summary

Tata Consultancy Services in Irving, TX is seeking a Data Engineer to design, build, and optimize scalable data pipelines using Spark, PySpark, and Hive in a cloud-first environment.

The role emphasizes data reliability, speed, and analytics readiness, collaborating with data scientists and analysts to translate business requirements into robust solutions. Experience with AWS/Azure/GCP data services is valued.

Qualifications

• Big data frameworks expertise with Spark architecture, drivers, executors, and DAGs.
• Advanced programming: Python and PySpark API for data transformations.
• Querying & schema management: HiveQL and ANSI SQL, with partitioning and schema definition.
• Optimized storage formats: Parquet, ORC, Avro.
• Cloud ecosystem development: AWS EMR, Azure Databricks, cloud data services.
• Data warehousing fundamentals: Star/Snowflake schemas and data lakes concepts.

Responsibilities

• Data Pipeline Development & Maintenance: Design, build, and maintain scalable ETL/ELT pipelines using PySpark and Spark SQL.
• Cloud Data Infrastructure Management: Deploy and scale data infrastructure on AWS, Azure, or GCP.
• Data Warehousing & Storage Optimization: Manage data layout, partitioning, indexing in Hive and cloud data lakes.
• Performance Tuning: Identify and resolve Spark job bottlenecks and memory issues.
• Diverse Data Integration: Ingest high-volume datasets from relational and unstructured sources.
• Automated Workflow Orchestration: Implement data workflows with Airflow or schedulers.
• Strategic Collaboration: Work with data scientists and analysts to translate requirements.

Skills

Apache Spark

PySpark API

Python

Spark SQL

HiveQL

ANSI SQL

Parquet/ORC/Avro

AWS EMR

Azure Databricks

Airflow

Data Modeling

Education

Bachelor's degree in Computer Science

Tools

AWS EMR

Azure Databricks

GCP (BigQuery)

Airflow

Job description

Tata Consultancy Services in Irving, TX is seeking a Data Engineer to design, build, and optimize scalable data pipelines using Spark, PySpark, and Hive in a cloud-first environment.

The role emphasizes data reliability, speed, and analytics readiness, collaborating with data scientists and analysts to translate business requirements into robust solutions. Experience with AWS/Azure/GCP data services is valued.