Azure Databricks Architect

Bilinguallink; Centre-du-Québec; On-site; CAD 120,000 - 180,000; Full time

Job summary

Bilinguallink is seeking an experienced Azure Databricks Architect to lead the design, implementation, and scaling of an enterprise data and AI platform in Canada. You will define the Lakehouse architecture, champion Unity Catalog governance, and spearhead performance tuning and cost optimization initiatives.

You will collaborate with data engineering, cloud infrastructure, security, and business stakeholders to deliver a robust, secure, and high-performance data ecosystem across Azure services.

Qualifications

• Experience designing enterprise Lakehouse architectures on Azure Databricks with Delta Lake.
• Deep understanding of Unity Catalog governance, RBAC/ABAC, and data security controls.
• Strong track record in performance tuning, cost optimization, and scalable data pipelines.

Responsibilities

• Lead Lakehouse architecture and design using Azure Databricks and Delta Lake.
• Architect secure, scalable data pipelines and Unity Catalog governance across environments.
• Drive performance tuning (AQE, partitioning) and cost optimization (autoscaling, spot compute).

Skills

Azure Databricks

Databricks Unity Catalog

Lakehouse Architecture

Performance Tuning

Cost Optimization

RBAC/ABAC

RLS

CLM

AQE

Z-ordering

Tools

Azure Data Lake Storage Gen2

Azure Data Factory

Azure Monitor

Spark

Job description

We are seeking an experienced and visionary Azure Databricks Architect to lead the design, implementation, and scaling of our enterprise data and AI platform. In this role, you will define the core architecture for our Lakehouse environment, champion governance standards using Databricks Unity Catalog, and spearhead end-to-end performance tuning and cost optimization initiatives. You will work closely with data engineering teams, cloud infrastructure teams, security officers, and business stakeholders to deliver a robust, secure, and high-performance data ecosystem.

Key Responsibilities

1. Lakehouse Architecture & Design

• Design and implement enterprise-grade Lakehouse architectures using Azure Databricks and Delta Lake, following Medallion Architecture (Bronze, Silver, Gold layers).
• Architect secure, scalable, and highly available cloud data pipelines supporting batch and real-time streaming workloads.
• Integrate Azure Databricks seamlessly with the broader Azure ecosystem (e.g., Azure Data Lake Storage Gen2, Azure Data Factory, Microsoft Entra ID, and Azure Monitor).

2. Data Governance & Security (Unity Catalog)

• Design, deploy, and manage Databricks Unity Catalog for centralized metadata management, data discovery, and lineage tracking across multi-workspace environments.
• Implement fine-grained access controls (RBAC/ABAC), row-level security (RLS), and column-level masking (CLM) to protect sensitive assets (PII/PHI).
• Establish auditing procedures, data sharing frameworks, and compliance guardrails using Unity Catalog audit logs.

3. Performance Tuning & Cost Optimization

• Drive Databricks and Spark optimization strategies, including managing shuffle partitions, resolving data skew, tuning cache, and applying Adaptive Query Execution (AQE).
• Implement layout strategies such as Z-ordering, data compaction (OPTIMIZE), and vacuum scheduling to curtail I/O overhead and boost query performance.
• Optimize cluster configurations, autoscaling parameters, spot instances, and serverless compute pools to maximize cost efficiency without compromising SLAs