Cloud Data Engineer: Spark & Hive Data Pipelines

Tata Consultancy Services Limited

Irving (TX)

On-site

USD 90,000 - 110,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Tata Consultancy Services in Irving, TX is seeking a Data Engineer to design, build, and optimize scalable data pipelines using Spark, PySpark, and Hive on the Cloudera platform. The role focuses on reliability, speed, and efficiency to support BI and advanced analytics initiatives.

You will develop ETL/ELT processes, optimize storage with Hive and cloud data lakes, and collaborate with data scientists and analysts to translate business requirements into robust data solutions.

Qualifications

  • Big Data Frameworks Expertise: Apache Spark architecture, drivers, executors, and DAGs.
  • Advanced Programming: Python and PySpark for complex data transformations.
  • Querying & Schema Management: HiveQL and ANSI SQL with partitioning and schema definition.
  • Optimized Storage Formats: Parquet, ORC, and Avro proficiency.
  • Data Warehousing Fundamentals: Dimensional modeling and Data Lakes concepts.

Responsibilities

  • Data Pipeline Development & Maintenance: Design, build, and maintain scalable ETL/ELT pipelines with PySpark and Spark SQL, Hive.
  • Data Warehousing & Storage Optimization: Manage data layout, partitioning, and indexing in Hive and cloud data lakes.
  • Performance Tuning & Optimization: Identify bottlenecks in Spark jobs, analyze with Spark UI, manage data skew and memory.
  • Diverse Data Integration: Ingest high-volume structured and unstructured data into the data ecosystem.
  • Automated Workflow Orchestration: Use Airflow or platform schedulers to ensure timely data delivery.
  • Strategic Collaboration: Work with data scientists, analysts, and cross-functional teams to translate requirements.

Skills

Apache Spark
PySpark
HiveQL
ANSI SQL
Python

Education

Bachelor's degree in Computer Science

Tools

Git
Jenkins
AWS EMR
Databricks
HBase
MongoDB

Job description

Tata Consultancy Services in Irving, TX is seeking a Data Engineer to design, build, and optimize scalable data pipelines using Spark, PySpark, and Hive on the Cloudera platform. The role focuses on reliability, speed, and efficiency to support BI and advanced analytics initiatives.

You will develop ETL/ELT processes, optimize storage with Hive and cloud data lakes, and collaborate with data scientists and analysts to translate business requirements into robust data solutions.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer: Cloudera, Spark & Hive Pipelines
Data Engineer: Cloudera, Spark & Hive Pipelines

Tata Consultancy Services Limited • Irving (TX)

On-site
USD 125,000 - 145,000
Data Engineer: Scalable Pipelines with Spark & Hive
Data Engineer: Scalable Pipelines with Spark & Hive

Tata Consultancy Services Limited • Irving (TX)

On-site
USD 125,000 - 140,000
Spark Data Engineer (PySpark) - Cloud ETL & Pipelines
Spark Data Engineer (PySpark) - Cloud ETL & Pipelines

Tata Consultancy Services Limited • Irving (TX)

On-site
USD 90,000 - 120,000
Discretionary Annual Incentive
Medical Coverage (Health, Dental & Vis
401K Plan
+1
AWS Data Engineer | Spark, PySpark, Hive & Airflow
AWS Data Engineer | Spark, PySpark, Hive & Airflow

Tata Consultancy Services Limited • Irving (TX)

On-site
USD 90,000 - 110,000
Remote Data Engineer: Cloud & Big Data Pipelines
Remote Data Engineer: Cloud & Big Data Pipelines

TheHiveCareers • United States

Remote
USD 30,000 - 60,000
Senior PySpark Data Engineer - Cloud & ETL Specialist
Senior PySpark Data Engineer - Cloud & ETL Specialist

Highbrow Technology Inc • Dallas (TX)

On-site
USD 140,000 - 200,000
Data Engineer: Snowflake, dbt & AWS Data Pipelines
Data Engineer: Snowflake, dbt & AWS Data Pipelines

Tata Consultancy Services Limited • Richardson (TX)

On-site
USD 130,000 - 145,000
Senior Data Engineer: PySpark, Airflow & GCP
Senior Data Engineer: PySpark, Airflow & GCP

Tata Consultancy Services Limited • Sunnyvale (CA)

On-site
USD 110,000 - 150,000
Discretionary Annual Incentive
Comprehensive Medical Coverage
Family Support: Parental Leaves
+4
Cloud Data Platform Engineer (Azure, PySpark, CDP)
Cloud Data Platform Engineer (Azure, PySpark, CDP)

Tata Consultancy Services Limited • San Jose (CA)

On-site
USD 80,000 - 115,000
Discretionary Annual Incentive
Comprehensive Medical Coverage
Family Support & Leaves
+1
Teradata Data Engineer: SQL, Spark & Data Pipelines
Teradata Data Engineer: SQL, Spark & Data Pipelines

Tata Consultancy Services Limited • Charlotte (NC)

On-site
USD 100,000 - 105,000
Discretionary Annual Incentive
Comprehensive Medical Coverage
401K Plan
+2