Big Data Engineer: Spark, PySpark & ETL Mastery

HCL Technologies Limited; Ciudad de México; Presencial; MXN 540.000 - 720.000; Jornada completa

Descripción de la vacante

HCLTech seeks a data engineering expert to design, build, and optimize large-scale data pipelines on Hadoop and cloud platforms like Databricks and Snowflake. You will work with Python, Spark, Hive, and Airflow to deliver robust data products and scalable ETL/ELT workflows.

You will collaborate with Product Managers to translate user needs into technical modules and contribute to DevOps practices to ensure reliable deployments.

Formación

• Production-grade pipelines on Hadoop and cloud data platforms.
• Strong Python, Spark, Hadoop, SQL skills.
• Experience with databricks and snowflake is preferred.
• Experience with orchestration tools like Airflow or NiFi.
• DevOps/CI-CD practices: Git, automated testing, release pipelines.

Responsabilidades

• Work with cutting-edge big data platforms at large scale to enable data processing and model development.
• Build and maintain ETL/ELT pipelines for ingestion, transformation, and aggregation of large-scale datasets on Hadoop and enterprise data platforms.
• Develop high-performance data processing jobs using PySpark/Spark on Cloudera and Databricks.
• Optimize pipeline performance and cost through partitioning and tuning.
• Contribute to CI/CD for data workflows and promote engineering best practices.
• Partner with Product Managers to understand users and use cases and scope new modules.

Conocimientos

Data engineering

Python

Spark

Hadoop

SQL

DevOps/CI-CD

Herramientas

Hive

Impala

Airflow

NiFi

Talend

Descripción del empleo

HCLTech seeks a data engineering expert to design, build, and optimize large-scale data pipelines on Hadoop and cloud platforms like Databricks and Snowflake. You will work with Python, Spark, Hive, and Airflow to deliver robust data products and scalable ETL/ELT workflows.

You will collaborate with Product Managers to translate user needs into technical modules and contribute to DevOps practices to ensure reliable deployments.