Cloud Data Engineer: Spark & Hive Data Pipelines

Tata Consultancy Services; Irving (TX); On-site; USD 90,000 - 110,000; Full time

Job summary

Tata Consultancy Services in Irving, TX is seeking a Data Engineer to design, build, and optimize scalable data pipelines using Spark, PySpark, and Hive on the Cloudera platform. The role focuses on reliability, speed, and efficiency to support BI and advanced analytics initiatives.

You will develop ETL/ELT processes, optimize storage with Hive and cloud data lakes, and collaborate with data scientists and analysts to translate business requirements into robust data solutions.

Qualifications

• Big Data Frameworks Expertise: Apache Spark architecture, drivers, executors, and DAGs.
• Advanced Programming: Python and PySpark for complex data transformations.
• Querying & Schema Management: HiveQL and ANSI SQL with partitioning and schema definition.
• Optimized Storage Formats: Parquet, ORC, and Avro proficiency.
• Data Warehousing Fundamentals: Dimensional modeling and Data Lakes concepts.

Responsibilities

• Data Pipeline Development & Maintenance: Design, build, and maintain scalable ETL/ELT pipelines with PySpark and Spark SQL, Hive.
• Data Warehousing & Storage Optimization: Manage data layout, partitioning, and indexing in Hive and cloud data lakes.
• Performance Tuning & Optimization: Identify bottlenecks in Spark jobs, analyze with Spark UI, manage data skew and memory.
• Diverse Data Integration: Ingest high-volume structured and unstructured data into the data ecosystem.
• Automated Workflow Orchestration: Use Airflow or platform schedulers to ensure timely data delivery.
• Strategic Collaboration: Work with data scientists, analysts, and cross-functional teams to translate requirements.

Skills

Apache Spark

PySpark

HiveQL

ANSI SQL

Python

Education

Bachelor's degree in Computer Science

Tools

Git

Jenkins

AWS EMR

Databricks

HBase

MongoDB

Job description

Tata Consultancy Services in Irving, TX is seeking a Data Engineer to design, build, and optimize scalable data pipelines using Spark, PySpark, and Hive on the Cloudera platform. The role focuses on reliability, speed, and efficiency to support BI and advanced analytics initiatives.

You will develop ETL/ELT processes, optimize storage with Hive and cloud data lakes, and collaborate with data scientists and analysts to translate business requirements into robust data solutions.