Senior Data Engineer - Spark, Databricks & CI/CD

HCL Technologies Limited; Vancouver; On-site; CAD 90,000 - 120,000; Full time

Job summary

HCL Technologies Limited in Vancouver seeks a data engineer to design and optimize large-scale data pipelines and platforms. You will work with Databricks, Spark, Python and SQL to deliver reliable ETL/ELT solutions, collaborating with product teams to translate user needs into scalable data products.

This role emphasizes DevOps practices, code quality, and cost-aware data processing in a modern cloud/Hadoop environment.

Qualifications

• Production-grade pipelines on Hadoop and cloud data platforms.
• Python, Spark, Hadoop tools and SQL for Big Data products.
• Cloud platforms such as Databricks and Snowflake (Databricks preferred).

Responsibilities

• Work with cutting-edge big data platforms at large scale to push data processing and model enablement.
• Build and maintain robust ETL/ELT pipelines for ingestion, transformation, and aggregation of large-scale datasets on Hadoop and enterprise data platforms.
• Develop high-performance data processing jobs using PySpark/Spark on platforms such as Cloudera and Databricks.
• Optimize pipeline performance and cost through partitioning, file formats, compute tuning, and efficient query patterns.
• Contribute to CI/CD for data workflows, testing, reviews, and deployment automation.
• Partner with Product Managers to understand users and use cases and translate into new modules and features.

Skills

Data Engineering

Databricks

Apache Spark

PySpark

Python

CI/CD

Tools

Hive

Impala

Airflow

NiFi

Git

Job description

HCL Technologies Limited in Vancouver seeks a data engineer to design and optimize large-scale data pipelines and platforms. You will work with Databricks, Spark, Python and SQL to deliver reliable ETL/ELT solutions, collaborating with product teams to translate user needs into scalable data products.

This role emphasizes DevOps practices, code quality, and cost-aware data processing in a modern cloud/Hadoop environment.