Azure Databricks Engineer

Data Elephant; Calgary; On-site; CAD 120,000 - 180,000; Full time

Job summary

Data Elephant is seeking an experienced Databricks Engineer to design, build, and optimize data platforms on Azure Databricks in Calgary on a contract basis. You will work with Data Elephant and client teams to migrate, modernize, and scale workloads while applying governance, ML, AI, and data security practices.

This hands-on role includes architecture decisions and end-to-end delivery. You will lead large data migrations, implement Delta Lake and Unity Catalog, and collaborate with analytics

Qualifications

• 7+ years of experience in data engineering.
• 4+ years of hands-on Databricks production solutions, Delta Lake, Unity Catalog, MLflow.
• Strong experience developing data pipelines using PySpark and SQL.
• Proven experience leading large-scale data migrations and cloud platform migrations.
• Strong expertise with Microsoft Azure and Azure data services (ADLS, ADF, Synapse, Key Vault, DevOps).
• Understanding of lakehouse/medallion architectures and ELT/ETL.
• Experience implementing CI/CD practices for data workloads.
• Strong governance, security, and compliance practices.
• Excellent communication with client teams; consulting experience preferred.
• Databricks and Azure certifications are assets.

Responsibilities

• Design and implement modern data platforms on Databricks (Azure) for migration and modernization.
• Build and optimize batch and streaming data workloads.
• Migrate legacy data workloads to Azure + Databricks.
• Integrate Databricks with ADLS, ADF, Synapse, Key Vault, and DevOps/GitHub.
• Apply governance models, Unity Catalog, data access control, and security standards.
• Collaborate with architects, analytics engineers, and client stakeholders.
• Develop reusable accelerators, standards, and best practices.
• Transfer knowledge and document delivery and architecture decisions.
• Provide hands-on solution delivery and client-facing support.
• Lead architecture discussions and ensure best practices in projects.

Skills

Data engineering

PySpark

SQL

Azure cloud

CI/CD

Education

Bachelor's degree in a related field

Tools

Databricks

Delta Lake

Unity Catalog

MLflow

Azure Data Factory

Job description

We are seeking an experienced Databricks Engineer to join our delivery team on a contract basis in Calgary to design, build, and optimize modern data platforms on Azure Databricks. In this strategic role, you will work directly with both Data Elephant and client data and analytics teams, helping them migrate, modernize, and scale their data workloads while applying best practices in data engineering, security, ML, AI and data governance. This is a hands-on engineering role with strong exposure to architecture decisions, client interaction, and end-to-end delivery.

Responsibilities

• Play an integral project team role in the implementation of modern data platforms and architectures on Databricks (including ETL, workload migrations, and Unity Catalog).
• Design and implement data pipelines using Azure Databricks (PySpark, SQL)
• Build and optimize batch and streaming data workloads
• Migrate legacy data workloads to Azure + Databricks
• Integrate Databricks with Azure services such as ADLS, ADF, Azure Key Vault and DevOps/Github
• Optimize performance and cost (cluster sizing, job orchestration, query tuning)
• Collaborate with solution architects, analytics engineers, and client stakeholders
• Contribute to reusable accelerators, standards, and internal best practices
• Support client enablement through knowledge transfer and documentation
• Provide hands-on solution delivery, including guiding and working closely with client engineers and ensuring best practices.
• Implement governance models and Unity Catalog including data access, lineage, and security frameworks.

QUALIFICATIONS

• 7+ years of experience in data engineering.
• 4+ years of recent, hands-on experience building production solutions with Databricks, including advanced features such as Delta Lake, Unity Catalog, and MLflow.
• Strong experience developing data pipelines using PySpark and SQL.
• Proven experience leading large-scale data migrations, including ETL workloads and cloud platform migrations.
• Strong expertise with Microsoft Azure and experience working with Azure data services, including ADLS, Azure Data Factory (ADF), Synapse Analytics, Azure Key Vault, and Azure DevOps.
• Strong understanding of modern data architectures, including lakehouse, medallion architecture, and ELT/ETL.
• Experience implementing CI/CD practices for data workloads.
• Strong understanding of enterprise data governance, security, and compliance best practices.
• Excellent communication skills with experience working directly with client teams.
• Previous consulting or professional services experience is strongly preferred.
• Databricks and Microsoft Azure certifications are considered an asset.

Be part of Canada’s leading boutique consulting firm focused on Databricks and modern data platforms.

• Work on challenging, high-impact projects with mid-sized and enterprise clients across industries
• Collaborate with a senior, high-performing team that values speed, pragmatism, and client success.