Principal Data Engineer – Databricks

Astra-North Infoteck Inc. ~ Conquering today’s challenges, achieving tomorrow’s vision!; Toronto; On-site; CAD 140,000 - 200,000; Full time

Job summary

Astra-North Infoteck Inc. in Toronto is seeking a Principal Data Engineer to lead Databricks, Spark, and Delta Lake initiatives within a Lakehouse framework. You will shape architecture for data platforms across cloud environments and guide senior engineers through complex transformations.

The role emphasizes scalable pipelines, governance, data lineage, security, and collaboration with finance, risk, analytics, and governance teams to translate data into business insights.

Qualifications

• 12–18 years in data engineering with strong analytics and architecture exposure.
• 8+ years in Enterprise Data Warehouse and Data Lake platforms.
• 5+ years with Databricks and Spark at scale in production environments.
• Experience migrating from Cloudera/Hive to Databricks Lakehouse.
• Strong data governance, lineage, and security controls.
• Proven ability to define metrics, dimensions, hierarchies and KPIs.

Responsibilities

• Lead Cloudera to Databricks transformation initiatives.
• Design and implement enterprise Data Lakehouse and Data Warehouse solutions.
• Build scalable, high‑performance data pipelines and architectures.
• Drive governance, quality, and security across platforms.
• Provide technical leadership, mentor teams, and establish standards.
• Support regulatory, management, and analytical reporting platforms.

Skills

Databricks

Apache Spark

Delta Lake

PySpark

Data Lakehouse

ETL pipelines

SQL

Data governance

CI/CD

Cloud (AWS/Azure)

Tools

Git

Terraform

Jenkins

Azure DevOps

Airflow

Databricks Workflows

dbt

Job description

Principal Data Engineer – Databricks | Spark | Delta Lake | PySpark | Data Lakehouse | AWS/Azure

Location: Toronto

Work Model: Onsite (4 days/week)

Key Requirements

• 12–18 years of overall Data Engineering experience.
• 8+ years of experience with Enterprise Data Warehouse and Data Lake platforms.
• 5+ years of hands‑on experience with Databricks and Apache Spark at scale.
• Strong experience modernizing legacy Cloudera platforms (CDH/CDP, Hive, HBase, Impala, Spark) to Databricks Lakehouse.
• Experience redesigning ingestion, transformation, and consumption patterns from HDFS-based architecture to cloud object storage and Delta Lake.
• Experience refactoring legacy Hive/Impala logic into PySpark and Spark SQL ELT pipelines.
• Experience ensuring data reconciliation, audit integrity, and consistency during migration.
• Experience designing and governing Enterprise Data Warehouse and Data Lake/Lakehouse architectures.
• Experience implementing layered architectures including:
  • Raw/Landing Layer
  • Curated/Conformed Layer
  • Semantic/Consumption Layer
• Experience modernizing traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures.
• Strong experience with finance and risk data models, including:
  • General Ledger
  • Sub‑ledger
  • Financial Hierarchies
  • Credit Risk Models
  • Liquidity Risk Models
  • Market Risk Models
• Experience enabling reporting use cases including aggregation, drill‑down, and drill‑back capabilities.
• Experience building and managing semantic/consumption layers for BI, reporting, and analytics.
• Ability to define business metrics, dimensions, hierarchies, and KPIs.
• Experience with Databricks SQL, Delta Tables, and dbt or similar frameworks.
• Strong experience developing and optimizing large‑scale data pipelines using:
  • PySpark
  • Spark SQL
  • Delta Lake
• Experience implementing Medallion Architecture:
  • Bronze Layer
  • Silver Layer
  • Gold Layer
• Experience optimizing workloads using Z‑ORDER, OPTIMIZE, caching, and cluster configurations.
• Experience implementing data governance, data quality frameworks, reconciliation controls, and exception handling.
• Experience establishing data lineage and metadata management.
• Knowledge of data security, access control, and compliance standards.
• Experience with cloud platforms such as AWS or Azure.
• Experience with CI/CD pipelines using:
  • Git
  • Terraform
  • Jenkins
  • Azure DevOps
• Familiarity with orchestration tools such as:
  • Apache Airflow
  • Databricks Workflows
• Experience with dbt is a plus.
• Ability to act as a technical authority and lead architecture decisions.
• Experience mentoring senior engineers and establishing engineering standards.
• Strong stakeholder management skills with finance, risk, analytics, and governance teams.
• Ability to translate complex data structures into business‑ready insights.

Nice to Have

• Experience in Banking, Financial Services, Insurance (BFSI), Capital Markets, or regulatory reporting.
• Exposure to:
  • SAP Finance
  • Oracle Financials
  • SAP S/4HANA
• Experience supporting AI/ML workloads.
• Databricks or cloud certifications.

Key Responsibilities

• Lead Cloudera to Databricks transformation initiatives.
• Design and implement enterprise Data Lakehouse and Data Warehouse solutions.
• Build scalable, high‑performance data pipelines and modern data architectures.
• Drive data modernization, governance, quality, and security initiatives.
• Support regulatory, management, and analytical reporting platforms.
• Provide technical leadership, mentor engineering teams, and establish best practices.