Data Engineer: Scalable Pipelines with Spark & Hive

Tata Consultancy Services; Irving (TX); On-site; USD 125,000 - 140,000; Full time

Job summary

Tata Consultancy Services in Irving, TX is seeking a Data Engineer to design, build, and optimize scalable data pipelines using Spark, PySpark, and Hive within Cloudera-like environments. Strong programming in Python, Spark SQL, and familiarity with Parquet/ORC formats are required, as is experience with Airflow or similar schedulers.

This role offers a competitive salary and opportunities to work on cutting-edge data platforms.

Qualifications

• Proficient in Spark architecture, drivers, executors, and DAGs.
• Strong Python and PySpark programming skills for complex transformations.
• Strong HiveQL/ANSI SQL with partitioning and schema design.
• Experience with Parquet/ORC/Avro storage formats.
• Solid foundation in dimensional modeling and data lake concepts.

Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines using PySpark and Spark SQL, Hive.
• Optimize data warehouse layouts, partitioning, and indexing for performance.
• Tune Spark jobs, monitor via Spark UI, and address memory issues and skew.
• Ingest high-volume structured and unstructured data into the data ecosystem.
• Automate data workflows with Airflow or platform schedulers for reliable delivery.
• Collaborate with data scientists and analysts to translate business requirements into data solutions.

Skills

Spark Architecture

Python / PySpark

HiveQL / ANSI SQL

Parquet/ORC/Avro

Dimensional Modeling

Education

Bachelor's degree in Computer Science

Tools

Git

Jenkins

Ansible

AWS EMR

Databricks

Job description

Tata Consultancy Services in Irving, TX is seeking a Data Engineer to design, build, and optimize scalable data pipelines using Spark, PySpark, and Hive within Cloudera-like environments. Strong programming in Python, Spark SQL, and familiarity with Parquet/ORC formats are required, as is experience with Airflow or similar schedulers.

This role offers a competitive salary and opportunities to work on cutting-edge data platforms.