Data Engineer: Cloudera, Spark & Hive Pipelines

Tata Consultancy Services Limited; Irving (TX); On-site; USD 125,000 - 145,000; Full time

Job summary

Tata Consultancy Services Limited in Irving, TX is seeking a data engineer to design and maintain scalable ETL/ELT pipelines using PySpark, Spark SQL, and Hive. You will optimize data storage and partitions while handling large, diverse datasets.

The role requires strong Python skills, HiveQL/SQL proficiency, and experience with Parquet/ORC/Avro formats. Collaboration with data scientists and analysts is essential.

Qualifications

• Extensive Spark architecture knowledge, drivers, executors, and DAGs.
• Strong Python coding skills with PySpark for transformations.
• Proficiency in HiveQL/SQL and data partitioning techniques.
• Experience with optimized big data storage formats (Parquet/ORC/Avro).
• Solid data warehousing foundation, dimensional modeling, data lakes.
• Databricks Certification is a plus.

Responsibilities

• Design, build and maintain scalable ETL/ELT pipelines with PySpark/Spark SQL and Hive.
• Optimize data warehousing layouts, partitioning, and indexing for performance.
• Tune Spark jobs and manage data skew and memory usage.
• Ingest large, diverse datasets from relational DBs and flat files.
• Automate workflows using Airflow or platform schedulers.
• Collaborate with data scientists and analysts to deliver data solutions.

Skills

Apache Spark

PySpark API

HiveQL / ANSI SQL

Data Modeling (Star/Snowflake)

Data Warehousing Concepts

Databricks Certification

Education

Bachelor of Computer Science

Tools

Spark / Core

Apache Hive

Parquet/ORC/Avro

AWS EMR / Databricks

Cloudera Hadoop (Hadoop Admin)

Job description

Tata Consultancy Services Limited in Irving, TX is seeking a data engineer to design and maintain scalable ETL/ELT pipelines using PySpark, Spark SQL, and Hive. You will optimize data storage and partitions while handling large, diverse datasets.

The role requires strong Python skills, HiveQL/SQL proficiency, and experience with Parquet/ORC/Avro formats. Collaboration with data scientists and analysts is essential.