Databricks Architect

Celebal Technologies; Canada; On-site; CAD 150,000 - 190,000; Full time

Job summary

Celebal Technologies is seeking an experienced Databricks Architect to design and implement scalable, secure, and high-performance data lakehouse solutions. The ideal candidate will master Databricks, Medallion Architecture, cloud platforms, SQL, PySpark, Python, and Databricks Auto Loader.

You will lead data architectures end‑to‑end, build streaming and batch pipelines, integrate with Azure or AWS, optimize Spark workloads, enforce governance and security, and mentor data engineering teams

Qualifications

• Experience designing Databricks lakehouse architectures and scalable data platforms.
• Hands-on with Medallion Architecture (Bronze/Silver/Gold) and Delta Lake.
• Strong SQL, PySpark, and Python skills for data processing.
• Proficient with Databricks Auto Loader and cloud integration (Azure/AWS).
• Knowledge of Unity Catalog, DLT, and data governance practices.
• Ability to lead technical teams and optimize production pipelines.

Responsibilities

• Design and implement enterprise-scale data architectures using Databricks.
• Develop scalable data pipelines with SQL, PySpark, and Python.
• Implement Medallion Architecture with Bronze, Silver, and Gold layers.
• Build incremental and streaming ingestion using Databricks Auto Loader.
• Integrate Databricks with Azure or AWS cloud services.
• Optimize Spark workloads, queries, and cluster configurations.
• Implement Delta Lake solutions for reliable storage and processing.
• Establish data governance, security, and access control best practices.
• Provide technical leadership and architectural guidance to data teams.
• Support production deployments, troubleshooting, and performance tuning.

Skills

Databricks architecture

Medallion Architecture

SQL

PySpark

Python

Databricks Auto Loader

Azure Databricks / AWS Databricks

Delta Lake

Data governance and security

Spark tuning & performance

Tools

Databricks Auto Loader

Unity Catalog

Delta Live Tables

Job description

Certification: Databricks Certification preferred

Job Summary

We are seeking an experienced Databricks Architect to design and implement scalable, secure, and high-performance data lakehouse solutions. The ideal candidate should have strong expertise in Databricks, Medallion Architecture, Azure/AWS cloud platforms, SQL, PySpark, Python, and Databricks Auto Loader.

Key Responsibilities

• Design and implement enterprise-scale data architectures using Databricks.
• Develop scalable data pipelines using SQL, PySpark, and Python.
• Implement Medallion Architecture, including Bronze, Silver, and Gold layers.
• Build incremental and streaming data ingestion pipelines using Databricks Auto Loader.
• Integrate Databricks with Azure or AWS cloud services.
• Optimize Spark workloads, SQL queries, cluster configurations, and data processing performance.
• Implement Delta Lake solutions for reliable data storage and processing.
• Establish data governance, security, and access control best practices.
• Provide technical leadership and architectural guidance to data engineering teams.
• Support production deployments, troubleshooting, and performance optimization.

Required Skills and Qualifications

• Strong experience with Databricks architecture and Lakehouse solutions.
• Hands-on expertise in Medallion Architecture.
• Proficiency in SQL, PySpark, and Python.
• Practical experience with Databricks Auto Loader.
• Experience with Azure Databricks or AWS Databricks.
• Strong understanding of Apache Spark, Delta Lake, and ETL/ELT pipelines.
• Experience in data pipeline optimization and production-grade data engineering.
• Knowledge of data governance, security, and scalable architecture design.
• Strong communication, analytical, and problem-solving skills.

Preferred Qualifications

• Databricks certification preferred.
• Experience with Unity Catalog and data governance.
• Knowledge of Delta Live Tables (DLT) or Lakeflow Declarative Pipelines.
• Experience with CI/CD, Git, and infrastructure-as-code tools.
• Experience with cloud migration and enterprise data platform modernization.
• Previous experience leading technical teams or defining data architecture standards.

Ideal Candidate

The ideal candidate will be a hands-on technical architect who can design end-to-end Databricks solutions, guide engineering teams, and deliver scalable data platforms using modern cloud and lakehouse technologies.