Databricks architect

HCLTech; Toronto; On-site; CAD 140,000 - 180,000; Full time

Job summary

HCLTech in Toronto seeks an senior Data Platform Architect to design and own a Databricks Lakehouse on Azure, with Delta Lake, Unity Catalog, and governance baked in. Stay hands-on by building PySpark pipelines, Delta Live Tables, notebooks, and orchestration, while setting the engineering bar and CI/CD standards with Azure DevOps.

You will lead migration of complex DataStage workloads to Databricks, optimize performance and cost, and partner with client leaders to translate requirements into

Qualifications

• 12+ years in data engineering / data platform architecture.
• 4+ years hands-on Databricks delivery experience.
• Strong governance, security, and cost-optimization focus.

Responsibilities

• Architect scalable, secure Databricks Lakehouse on Azure.
• Lead migration of legacy ETL workloads to Databricks and ADF.
• Coach data engineers and set CI/CD standards with Azure DevOps.
• Collaborate with client leadership to translate requirements into delivery roadmaps.

Skills

Databricks

PySpark

Spark SQL

Delta Lake

Unity Catalog

Azure Data Factory

Azure DevOps

CI/CD

Security & Governance

Tools

Delta Live Tables

Purview

MLflow

Azure ML

ADF

Job description

Architect the lakehouse: Design and own scalable, secure Databricks Lakehouse architecture on Azure (Delta Lake, Unity Catalog, medallion bronze/silver/gold, ADLS Gen2) aligned to enterprise standards.

Stay hands-on: Personally build and review PySpark / Spark SQL pipelines, Delta Live Tables, notebooks, and orchestration — setting the engineering bar, not just directing it.

Lead legacy migration: Drive conversion of complex legacy ETL (DataStage) workloads to Databricks/PySpark and ADF, including patterns, accelerators, and reusable frameworks for code conversion and validation.

Own performance & cost: Optimize cluster configuration, job performance, partitioning, and cost; establish FinOps and right-sizing practices on Databricks.

Embed governance: Implement data governance, lineage, quality, and access control through Unity Catalog and Purview; ensure security, privacy, and compliance by design.

Enable analytics & AI: Design Gold-layer semantic models and feature pipelines that serve BI (Power BI), advanced analytics, and ML/GenAI use cases (MLflow, Azure ML).

Lead the squad: Provide technical leadership and mentoring to data engineers; define best practices, coding standards, CI/CD (Azure DevOps), and review processes.

Partner with the client: Work closely with the client’s VP (Data & AI), AVP (Data Platforms & Integration), platform architects, and business stakeholders to translate requirements into delivery roadmaps and measurable outcomes.

Required Qualifications (Must-Have)

12+ years in data engineering / data platform architecture, with 4+ years of deep, hands-on Databricks delivery.