Data Engineer

HCL Technologies Limited; Vancouver; Presencial; CAD 90.000 - 120.000; Jornada completa

Descripción de la vacante

HCL Technologies Limited in Vancouver seeks a data engineer to design and optimize large-scale data pipelines and platforms. You will work with Databricks, Spark, Python and SQL to deliver reliable ETL/ELT solutions, collaborating with product teams to translate user needs into scalable data products.

This role emphasizes DevOps practices, code quality, and cost-aware data processing in a modern cloud/Hadoop environment.

Formación

• Production-grade pipelines on Hadoop and cloud data platforms.
• Python, Spark, Hadoop tools and SQL for Big Data products.
• Cloud platforms such as Databricks and Snowflake (Databricks preferred).

Responsabilidades

• Work with cutting-edge big data platforms at large scale to push data processing and model enablement.
• Build and maintain robust ETL/ELT pipelines for ingestion, transformation, and aggregation of large-scale datasets on Hadoop and enterprise data platforms.
• Develop high-performance data processing jobs using PySpark/Spark on platforms such as Cloudera and Databricks.
• Optimize pipeline performance and cost through partitioning, file formats, compute tuning, and efficient query patterns.
• Contribute to CI/CD for data workflows, testing, reviews, and deployment automation.
• Partner with Product Managers to understand users and use cases and translate into new modules and features.

Conocimientos

Data Engineering

Databricks

Apache Spark

PySpark

Python

CI/CD

Herramientas

Hive

Impala

Airflow

NiFi

Git

Descripción del empleo

Work with cutting-edge big data platforms (e.g., Databricks, Apache Spark) at large scale, pushing the boundaries of data processing and model enablement.

Build and maintain robust ETL/ELT pipelines for ingestion, transformation, and aggregation of large-scale datasets on Hadoop and enterprise data platforms.

Develop high-performance data processing jobs using PySpark/Spark, Python on data platforms such as cloudera and databricks.

Optimize pipeline performance and cost through partitioning, file formats, compute tuning, and efficient query patterns

Contribute to CI/CD for data workflows (testing, code reviews, deployment automation), promoting engineering best practices and maintainable codebases.

Partner with Product Managers to develop a deep understanding of users and use cases and apply that knowledge to scoping and building new modules and features

Ideal Candidate Qualifications:

Strong hands‑on experience in data engineering building production‑grade pipelines on big data platforms (Hadoop ecosystem and cloud data platforms - databricks).

High proficiency in using Python, Spark, Hadoop platforms & tools (Hive, Impala, Airflow, NiFi), SQL to build Big Data products.

Hands‑on experience with cloud data platforms such as databricks, snowflake (databricks preferred)

Experience with orchestration/integration tools such as Apache Airflow, Apache NiFi, or Talend.

Working knowledge of DevOps/CI-CD practices: version control (Git), automated testing, release pipelines, and observability.

Strong problem‑solving skills with the ability to debug complex data issues and communicate clearly with technical and non‑technical stakeholders.

Experience developing Java based applications is an added advantage.

LONGDESCRIPTION section. 2 of 6.

Key Responsibilities

Work with cutting-edge big data platforms (e.g., Databricks, Apache Spark) at large scale, pushing the boundaries of data processing and model enablement.

Build and maintain robust ETL/ELT pipelines for ingestion, transformation, and aggregation of large-scale datasets on Hadoop and enterprise data platforms.

Develop high-performance data processing jobs using PySpark/Spark, Python on data platforms such as cloudera and databricks.

Optimize pipeline performance and cost through partitioning, file formats, compute tuning, and efficient query patterns

Contribute to CI/CD for data workflows (testing, code reviews, deployment automation), promoting engineering best practices and maintainable codebases.

Partner with Product Managers to develop a deep understanding of users and use cases and apply that knowledge to scoping and building new modules and features

Ideal Candidate Qualifications:

Strong hands‑on experience in data engineering building production‑grade pipelines on big data platforms (Hadoop ecosystem and cloud data platforms - databricks).

High proficiency in using Python, Spark, Hadoop platforms & tools (Hive, Impala, Airflow, NiFi), SQL to build Big Data products.

Hands‑on experience with cloud data platforms such as databricks, snowflake (databricks preferred)

Experience with orchestration/integration tools such as Apache Airflow, Apache NiFi, or Talend.

Working knowledge of DevOps/CI-CD practices: version control (Git), automated testing, release pipelines, and observability.

Strong problem‑solving skills with the ability to debug complex data issues and communicate clearly with technical and non‑technical stakeholders.

Experience developing Java based applications is an added advantage.

LONGDESCRIPTION section. 3 of 6.

Skill Requirements

Work with cutting-edge big data platforms (e.g., Databricks, Apache Spark) at large scale, pushing the boundaries of data processing and model enablement.

Build and maintain robust ETL/ELT pipelines for ingestion, transformation, and aggregation of large-scale datasets on Hadoop and enterprise data platforms.

Develop high-performance data processing jobs using PySpark/Spark, Python on data platforms such as cloudera and databricks.

Optimize pipeline performance and cost through partitioning, file formats, compute tuning, and efficient query patterns

Contribute to CI/CD for data workflows (testing, code reviews, deployment automation), promoting engineering best practices and maintainable codebases.

Partner with Product Managers to develop a deep understanding of users and use cases and apply that knowledge to scoping and building new modules and features

Ideal Candidate Qualifications:

Strong hands‑on experience in data engineering building production‑grade pipelines on big data platforms (Hadoop ecosystem and cloud data platforms - databricks).

High proficiency in using Python, Spark, Hadoop platforms & tools (Hive, Impala, Airflow, NiFi), SQL to build Big Data products.

Hands‑on experience with cloud data platforms such as databricks, snowflake (databricks preferred)

Experience with orchestration/integration tools such as Apache Airflow, Apache NiFi, or Talend.

Working knowledge of DevOps/CI-CD practices: version control (Git), automated testing, release pipelines, and observability.

Strong problem‑solving skills with the ability to debug complex data issues and communicate clearly with technical and non‑technical stakeholders.

Experience developing Java based applications is an added advantage.

SKILL section. 4 of 6.

Must Have Skills

• Click Enter to show the proficiency description of Data EngineeringData Engineering
• Click Enter to show the proficiency description of DatabricksDatabricks
• Click Enter to show the proficiency description of Apache SparkApache Spark
• Click Enter to show the proficiency description of PySparkPySpark
• Click Enter to show the proficiency description of PythonPython
• Click Enter to show the proficiency description of CI/CDCI/CD

SKILL section. 5 of 6.

Good to have Skills

LONGDESCRIPTION section. 6 of 6.

Other Requirements

Work with cutting-edge big data platforms (e.g., Databricks, Apache Spark) at large scale, pushing the boundaries of data processing and model enablement.

Build and maintain robust ETL/ELT pipelines for ingestion, transformation, and aggregation of large-scale datasets on Hadoop and enterprise data platforms.

Develop high-performance data processing jobs using PySpark/Spark, Python on data platforms such as cloudera and databricks.

Optimize pipeline performance and cost through partitioning, file formats, compute tuning, and efficient query patterns

Contribute to CI/CD for data workflows (testing, code reviews, deployment automation), promoting engineering best practices and maintainable codebases.

Partner with Product Managers to develop a deep understanding of users and use cases and apply that knowledge to scoping and building new modules and features

Ideal Candidate Qualifications:

Strong hands-on experience in data engineering building production-grade pipelines on big data platforms (Hadoop ecosystem and cloud data platforms - databricks).

High proficiency in using Python, Spark, Hadoop platforms & tools (Hive, Impala, Airflow, NiFi), SQL to build Big Data products.

Hands‑on experience with cloud data platforms such as databricks, snowflake (databricks preferred)

Experience with orchestration/integration tools such as Apache Airflow, Apache NiFi, or Talend.

Working knowledge of DevOps/CI-CD practices: version control (Git), automated testing, release pipelines, and observability.

Strong problem‑solving skills with the ability to debug complex data issues and communicate clearly with technical and non‑technical stakeholders.

Experience developing Java based applications is an added advantage