Principal Data Engineer – Databricks

Astra-North Infoteck Inc.

Toronto

On-site

CAD 140,000 - 190,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Astra-North Infoteck Inc. in Toronto seeks a Principal Data Engineer to lead the Databricks, Spark, and Delta Lake migration to a modern data lakehouse. The role focuses on scalable pipelines, governance, and secure data architectures across AWS/Azure environments.

You will drive architecture decisions, mentor senior engineers, and collaborate with finance, risk, and governance teams to deliver business-ready insights.

Qualifications

  • 12–18 years of overall Data Engineering experience.
  • 8+ years of Enterprise Data Warehouse and Data Lake platforms experience.
  • 5+ years of hands-on experience with Databricks and Apache Spark at scale.
  • Strong experience modernizing legacy Cloudera platforms (CDH/CDP, Hive, HBase, Impala, Spark) to Databricks Lakehouse.
  • Experience redesigning ingestion, transformation, and consumption patterns from HDFS-based architecture to cloud object storage and Delta Lake.
  • Experience refactoring legacy Hive/Impala logic into PySpark and Spark SQL ELT pipelines.
  • Experience ensuring data reconciliation, audit integrity, and consistency during migration.
  • Experience designing and governing Enterprise Data Warehouse and Data Lake/Lakehouse architectures.
  • Experience implementing layered architectures including: Curated/Conformed Layer.
  • Experience modernizing traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures.
  • Strong experience with finance and risk data models, including: Financial Hierarchies, Credit Risk Models, Liquidity Risk Models.
  • Experience enabling reporting use cases including aggregation, drill-down, and drill-back capabilities.
  • Experience building and managing semantic/consumption layers for BI, reporting, and analytics.
  • Ability to define business metrics, dimensions, hierarchies, and KPIs.
  • Experience with Databricks SQL, Delta Tables, and dbt or similar frameworks.
  • Strong experience developing and optimizing large-scale data pipelines using: Experience optimizing workloads using Z-ORDER, OPTIMIZE, caching, and cluster configurations.
  • Experience implementing data governance, data quality frameworks, reconciliation controls, and exception handling.
  • Experience establishing data lineage and metadata management.
  • Knowledge of data security, access control, and compliance standards.
  • Experience with cloud platforms such as AWS or Azure.
  • Experience with CI/CD pipelines using:
  • Familiarity with orchestration tools such as:
  • Experience with dbt is a plus.
  • Ability to act as a technical authority and lead architecture decisions.
  • Experience mentoring senior engineers and establishing engineering standards.
  • Strong stakeholder management skills with finance, risk, analytics, and governance teams.
  • Ability to translate complex data structures into business-ready insights.

Responsibilities

  • Lead Cloudera to Databricks transformation initiatives.
  • Design and implement enterprise Data Lakehouse and Data Warehouse solutions.
  • Build scalable, high-performance data pipelines and modern data architectures.
  • Drive data modernization, governance, quality, and security initiatives.
  • Support regulatory, management, and analytical reporting platforms.
  • Provide technical leadership, mentor engineering teams, and establish best practices.

Skills

Databricks
Apache Spark
Delta Lake
PySpark
Data Lakehouse
AWS/Azure
dbt
SQL
Data Governance
Data Quality
CI/CD

Tools

Databricks
dbt
Delta Tables

Job description

Principal Data Engineer – Databricks | Spark | Delta Lake | PySpark | Data Lakehouse | AWS/Azure
Job Description

Location: Toronto

Work Model: Onsite (4 days/week)

Key Requirements
  • 12–18 years of overall Data Engineering experience.
  • 8+ years of experience with Enterprise Data Warehouse and Data Lake platforms.
  • 5+ years of hands‑on experience with Databricks and Apache Spark at scale.
  • Strong experience modernizing legacy Cloudera platforms (CDH/CDP, Hive, HBase, Impala, Spark) to Databricks Lakehouse.
  • Experience redesigning ingestion, transformation, and consumption patterns from HDFS-based architecture to cloud object storage and Delta Lake.
  • Experience refactoring legacy Hive/Impala logic into PySpark and Spark SQL ELT pipelines.
  • Experience ensuring data reconciliation, audit integrity, and consistency during migration.
  • Experience designing and governing Enterprise Data Warehouse and Data Lake/Lakehouse architectures.
  • Experience implementing layered architectures including:
    • Curated/Conformed Layer
  • Experience modernizing traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures.
  • Strong experience with finance and risk data models, including:
    • Financial Hierarchies
    • Credit Risk Models
    • Liquidity Risk Models
  • Experience enabling reporting use cases including aggregation, drill-down, and drill-back capabilities.
  • Experience building and managing semantic/consumption layers for BI, reporting, and analytics.
  • Ability to define business metrics, dimensions, hierarchies, and KPIs.
  • Experience with Databricks SQL, Delta Tables, and dbt or similar frameworks.
  • Strong experience developing and optimizing large-scale data pipelines using:
    • Experience optimizing workloads using Z-ORDER, OPTIMIZE, caching, and cluster configurations.
  • Experience implementing data governance, data quality frameworks, reconciliation controls, and exception handling.
  • Experience establishing data lineage and metadata management.
  • Knowledge of data security, access control, and compliance standards.
  • Experience with cloud platforms such as AWS or Azure.
  • Experience with CI/CD pipelines using:
  • Familiarity with orchestration tools such as:
  • Experience with dbt is a plus.
  • Ability to act as a technical authority and lead architecture decisions.
  • Experience mentoring senior engineers and establishing engineering standards.
  • Strong stakeholder management skills with finance, risk, analytics, and governance teams.
  • Ability to translate complex data structures into business-ready insights.
Nice to Have
  • Experience in Banking, Financial Services, Insurance (BFSI), Capital Markets, or regulatory reporting.
  • Oracle Financials
  • Experience supporting AI/ML workloads.
  • Databricks or cloud certifications.
Key Responsibilities
  • Lead Cloudera to Databricks transformation initiatives.
  • Design and implement enterprise Data Lakehouse and Data Warehouse solutions.
  • Build scalable, high-performance data pipelines and modern data architectures.
  • Drive data modernization, governance, quality, and security initiatives.
  • Support regulatory, management, and analytical reporting platforms.
  • Provide technical leadership, mentor engineering teams, and establish best practices.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Principal Databricks Data Engineer
Principal Databricks Data Engineer

Astra-North Infoteck Inc. • Toronto

On-site
CAD 150,000 - 190,000
Senior Databricks Data Engineer
Senior Databricks Data Engineer

Raise - find a more meaningful working experience • Toronto

Hybrid
CAD 100,000 - 167,000
Senior Databricks Engineer
Senior Databricks Engineer

KData Inc. • Toronto

On-site
CAD 170,000 - 210,000
Databricks Architect
Databricks Architect

TechDoQuest • Toronto

On-site
CAD 100,000 - 130,000
Databricks Engineer
Databricks Engineer

Compunnel Inc. • Mississauga

On-site
CAD 120,000 - 150,000
Senior AWS Data Engineer (Strong Databricks)
Senior AWS Data Engineer (Strong Databricks)

Ccube • Mississauga

Hybrid
CAD 120,000 - 170,000
Senior Databricks Engineer
Senior Databricks Engineer

KData AI • Toronto

On-site
CAD 130,000 - 180,000
Data Engineer
Data Engineer

Spectraforce • Toronto

Hybrid
CAD 80,000 - 110,000
Lead/Senior Data Engineer
Lead/Senior Data Engineer

Celebal Technologies • Ottawa

On-site
CAD 110,000 - 160,000
Databricks Data Platform Engineer
Databricks Data Platform Engineer

Mindtris • Canada

Remote
CAD 120,000 - 180,000