Principal Databricks Data Engineer

Astra-North Infoteck Inc.

Toronto

On-site

CAD 150,000 - 190,000

Full time

14 days+

Get a reply from this recruiter

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Astra-North Infoteck Inc. is seeking an accomplished Data Engineering/Architecture lead in Toronto to drive the Cloudera to Databricks transformation. You will shape finance and risk data platforms and enable regulatory, management, and analytics reporting across the organization.

The role requires deep expertise in Databricks, Spark, data lakehouse architectures, and cloud platforms (AWS/Azure). You will mentor engineers, establish standards, and collaborate with governance stakeholders to

Qualifications

  • 12–18 years of overall Data Engineering experience.
  • 8+ years of enterprise Data Warehouse and Data Lake platform experience.
  • 5+ years hands-on with Databricks and Apache Spark at scale.
  • Modernize Cloudera platforms to Databricks Lakehouse architecture.
  • Redesign ingestion, transformation, and consumption from HDFS-based architecture to Cloud Object Storage and Delta Lake.
  • Refactor Hive and Impala logic into PySpark and Spark SQL ELT pipelines.
  • Ensure data reconciliation, audit integrity, and consistency during migration.
  • Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures.
  • Strong finance and risk data models including Financial Hierarchies and Liquidity Risk.
  • Enable reporting: semantic/consumption layers, define metrics, dimensions, KPIs.
  • Experience with AWS or Azure cloud platforms; CI/CD; data security and governance.
  • dbt is an advantage.

Responsibilities

  • Lead Cloudera to Databricks transformation initiatives.
  • Shape enterprise finance and risk data platforms.
  • Support regulatory, management, and analytical reporting systems.

Skills

Databricks
Apache Spark
Cloudera modernization
Data Warehouse
Data Lake
Delta Lake
HDFS to Cloud
PySpark
Spark SQL ELT
Data lineage
CI/CD pipelines
Security & compliance
Cloud platforms
dbt
Oracle Financials

Tools

dbt
Oracle Financials

Job description

  • 12–18 years of overall Data Engineering experience
  • 8+ years of experience with enterprise Data Warehouse and Data Lake platforms
  • 5+ years of hands‑on experience with Databricks and Apache Spark at scale
  • Strong experience modernizing legacy Cloudera platforms including:
    • Modernize Cloudera platforms to Databricks Lakehouse architecture
    • Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to Cloud Object Storage and Delta Lake
    • Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines
    • Ensure data reconciliation, audit integrity, and consistency during migration
    • Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures
    • Curated / Conformed Layer
    • Semantic / Consumption Layer
    • Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures
  • Strong experience with finance and risk data models including:
    • Financial Hierarchies
    • Liquidity Risk
  • Enable reporting capabilities including:
    • Build and manage semantic and consumption layers for BI, reporting, and analytics
    • Define business metrics, dimensions, hierarchies, and KPIs
  • Experience with:
    • Develop and optimize large‑scale data pipelines using:
    • Optimize workloads using:
    • Data Quality Frameworks
    • Establish data lineage and metadata management
    • Ensure data security, access control, and compliance standards
    • Experience with AWS or Azure cloud platforms
    • Experience with CI/CD pipelines using:
    • Familiarity with:
    • Experience with dbt is an advantage
  • Act as a technical authority and lead enterprise architecture decisions
  • Mentor senior engineers and establish engineering standards
  • Collaborate with finance, risk, analytics, and governance stakeholders
  • Translate complex data structures into business‑ready insights
Nice‑to‑Have Skills
  • Experience in BFSI, Capital Markets, or Regulatory Reporting
  • Oracle Financials
  • Experience supporting AI/ML workloads
  • Databricks and Cloud Certifications
Key Responsibilities
  • Lead Cloudera to Databricks transformation initiatives
  • Shape enterprise finance and risk data platforms
  • Support regulatory, management, and analytical reporting systems
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Principal Data Engineer – Databricks
Principal Data Engineer – Databricks

Astra-North Infoteck Inc. • Toronto

On-site
CAD 140,000 - 190,000
Databricks Architect
Databricks Architect

TechDoQuest • Toronto

On-site
CAD 100,000 - 130,000
Databricks Engineer
Databricks Engineer

Compunnel Inc. • Mississauga

On-site
CAD 120,000 - 150,000
Senior Databricks Engineer
Senior Databricks Engineer

KData AI • Toronto

On-site
CAD 130,000 - 180,000
Senior Databricks Resident Solutions Architect
Senior Databricks Resident Solutions Architect

Sohogtech • Canada

Remote
CAD 212,000 - 283,000
Senior Databricks Data Engineer
Senior Databricks Data Engineer

Raise - find a more meaningful working experience • Toronto

Hybrid
CAD 100,000 - 167,000
Senior Databricks Engineer
Senior Databricks Engineer

KData Inc. • Toronto

On-site
CAD 170,000 - 210,000
Databricks Engineer - Azure Data Pipelines & Lakehouse
Databricks Engineer - Azure Data Pipelines & Lakehouse

Compunnel Inc. • Mississauga

On-site
CAD 120,000 - 150,000
Databricks Data Engineer
Databricks Data Engineer

Rubicon Path • Toronto

On-site
CAD 80,000 - 110,000
Data Science Developer - Senior JP210
Data Science Developer - Senior JP210

P@thlion Staffing Careers • Fredericton

On-site
CAD 70,000 - 110,000