Sr Databricks Data Engineer

Pacer Group; Toronto; On-site; CAD 110,000 - 170,000; Full time

Job summary

Pacer Group in Toronto, Canada is seeking a data engineering lead to advance analytics platforms. You will migrate legacy Cloudera environments to Databricks Lakehouse, design scalable Data Warehouse and Lakehouse architectures, and optimize for performance and cost.

You will develop high-volume PySpark/Spark SQL ELT pipelines, work with Finance and Risk data models, and collaborate with cross-functional teams to deliver robust data solutions.

Qualifications

• Hands-on experience with modern data platforms and abstractions.
• Experience migrating from legacy Cloudera (CDH/CDP, Hive, HBase, Impala, Spark) to Databricks Lakehouse.
• Strong knowledge of Data Warehouse, Data Lake/Lakehouse, Delta Lake, and Medallion architectures.

Responsibilities

• Lead modernization of Cloudera platforms to Databricks Lakehouse.
• Design and implement scalable data warehouse, data lake/lakehouse architectures.
• Develop high-volume PySpark/Spark SQL ELT pipelines and optimize Databricks workloads.

Skills

Databricks

Apache Spark

PySpark

Spark SQL

Data Modeling

Data Lakehouse

Delta Lake

Medallion Architecture

Cost Optimization

Tools

Databricks

Apache Spark

PySpark

Spark SQL

Delta Lake

Job description

• Hands-on experience with Databricks and Apache Spark at scale.
• Lead modernization of legacy Cloudera (CDH/CDP, Hive, HBase, Impala, Spark) platforms to Databricks Lakehouse.
• Design and implement scalable Data Warehouse, Data Lake/Lakehouse, Delta Lake, and Medallion architectures.
• Develop high-volume PySpark/Spark SQL ELT pipelines and optimize Databricks workloads for performance and cost.
• Strong experience with Finance and Risk data models, including General Ledger, accounting, credit, liquidity, market risk, and regulatory reporting.

EEOC Compliance:

We are an equal opportunity employer, and all qualified applicants will receive consideration for employment.

DISCLAIMER

AI Usage Policy: Pacer Group uses AI to assist in screening applications. Final hiring decisions are made by human recruiters based on qualifications and experience.