AWS Data Engineer | Spark, PySpark, Hive & Airflow

Tata Consultancy Services Limited; Irving (TX); Sur place; USD 90 000 - 110 000; Plein temps

Résumé du poste

Tata Consultancy Services Limited in Irving, TX is seeking a Data Engineer to design, build, and optimize scalable data pipelines using PySpark, Spark SQL, and Hive on Cloudera. You will ensure data reliability, speed, and efficiency to support BI and analytics initiatives.

You will collaborate with data scientists and analysts, develop ETL/ELT processes, optimize data warehousing, and automate workflows with Airflow, leveraging Python and SQL skills to deliver robust data solutions.

Qualifications

• Experience with Spark architecture, DAGs, and Spark UI.
• Strong Python and PySpark for complex transformations.
• Proficient in HiveQL/ANSI SQL and partitioning techniques.
• Familiarity with Parquet/ORC/Avro formats.
• Solid data warehousing and data lake concepts.

Responsabilités

• Data Pipeline Development & Maintenance: design, build, and maintain scalable ETL/ELT pipelines using PySpark, Spark SQL, Hive.
• Data Warehousing & Storage Optimization: manage layout, partitioning, and indexing in Hive and cloud data lakes.
• Performance Tuning & Optimization: identify bottlenecks in Spark jobs and optimize memory usage.
• Diverse Data Integration: ingest high-volume structured and unstructured data into the data ecosystem.
• Automated Workflow Orchestration: implement workflows with Airflow or platform schedulers.
• Strategic Collaboration: work with data scientists and analysts to translate requirements into solutions.

Connaissances

PySpark

Spark SQL

Hive

Python

SQL

Partitioning

Data Lakes

Dimensional Modeling

Airflow

Data Pipelines

Formation

Bachelor's degree in Computer Science

Outils

Airflow

Git

Jenkins

Ansible

AWS EMR

Databricks

Description du poste

Tata Consultancy Services Limited in Irving, TX is seeking a Data Engineer to design, build, and optimize scalable data pipelines using PySpark, Spark SQL, and Hive on Cloudera. You will ensure data reliability, speed, and efficiency to support BI and analytics initiatives.

You will collaborate with data scientists and analysts, develop ETL/ELT processes, optimize data warehousing, and automate workflows with Airflow, leveraging Python and SQL skills to deliver robust data solutions.