Sr. Azure Data Engineer-DataBricks

BeachHead Inc.; Toronto; On-site; CAD 120,000 - 180,000; Full time

Job summary

BeachHead Inc. is seeking a Sr. Azure Data Engineer-DataBricks to join a dynamic team supporting large-scale data solutions in the asset management sector. The role emphasizes advanced Databricks implementations, data governance, and cloud-based data platforms to enable strategic decisions.

The candidate will develop ADF orchestrations with Databricks, manage Unity Catalog, and design Delta Lake schemas while ensuring security and compliance in a regulated financial services environment.

Qualifications

• 7+ years with Azure Data Factory, Databricks and Azure Functions (C#).
• Python, notebooks, Spark and pandas in Databricks expertise.
• Unity Catalog and data governance setup in a cloud environment.
• Experience loading datasets from various sources using ADF in a Medallion architecture.
• Proficiency managing Databricks workspace permissions and security.
• Delta Live Tables, cost and performance optimization knowledge.
• DevOps pipelines using Azure DevOps or GitHub Enterprise for CI/CD.
• Ability to work in a highly regulated financial services environment.

Responsibilities

• Develop and maintain Azure Data Factory orchestrations with Databricks integrations
• Configure and manage Unity Catalog for data governance and access control
• Design and implement Delta Lake schemas and tables with appropriate security permissions
• Load datasets from diverse sources into Databricks using ADF with Medallion architecture
• Convert Databricks workspace from hive metastore to Unity Catalog and set up associated permissions
• Build and optimize Delta Live Tables, ensuring performance and cost-efficiency
• Develop deployment pipelines using Azure DevOps and GitHub Enterprise for CI/CD processes
• Deploy and manage Azure PaaS resources such as Azure SQL, Azure Data Lake, and Synapse
• Collaborate with team members independently and provide technical leadership
• Support a highly regulated environment with a focus on data security, governance, and compliance
• Troubleshoot and resolve complex data platform issues efficiently

Skills

Azure Data Factory

Databricks

Azure Functions (C#)

Python

Spark

Pandas

Unity Catalog

Medallion architecture

Delta Live Tables

Azure DevOps

GitHub Enterprise

Tools

Azure Data Factory

Databricks

Azure DevOps

GitHub Enterprise

Synapse

Azure SQL

Azure Data Lake

Delta Lake

Job description

Working with one of our top financial clients, this role calls for a Sr. Azure Data Engineer-DataBricks. This position involves working within a dynamic team to support large-scale data solutions in the asset management sector, focusing on advanced Databricks implementations, data governance, and cloud-based data platforms to enable strategic decision-making.

Responsibilities

• Develop and maintain Azure Data Factory orchestrations with Databricks integrations
• Configure and manage Unity Catalog for data governance and access control
• Design and implement Delta Lake schemas and tables with appropriate security permissions
• Load datasets from diverse sources into Databricks using ADF with Medallion architecture
• Convert Databricks workspace from hive metastore to Unity Catalog and set up associated permissions
• Build and optimize Delta Live Tables, ensuring performance and cost-efficiency
• Develop deployment pipelines using Azure DevOps and GitHub Enterprise for CI/CD processes
• Deploy and manage Azure PaaS resources such as Azure SQL, Azure Data Lake, and Synapse
• Collaborate with team members independently and provide technical leadership
• Support a highly regulated environment with a focus on data security, governance, and compliance
• Troubleshoot and resolve complex data platform issues efficiently

Desired Skill-Set

• 7+ years' experience with Azure Data Factory, Databricks, and Azure Functions (C#)
• Deep knowledge of Python, notebooks, Spark, and pandas within Databricks
• Expertise in Unity Catalog and data governance setup in a cloud environment
• Experience loading datasets from various sources using ADF in a Medallion architecture
• Proficiency in converting and managing Databricks workspace permissions and security
• Familiarity with Delta Live Tables, cost, and performance optimization
• Experience with Azure DevOps or GitHub for DevOps pipelines and CI/CDWorking knowledge of Azure Entra ID for SSO and RBAC configurationAbility to work in a highly regulated industry environment with limited accessStrong communication skills and ability to lead small teams independentlyFinancial services industry experience is an asset

  Nice to Have

  • Previous experience working withfinancial institutions

  BeachHead is an equal opportunity agency and employer. We advocate for our candidates and welcome applicants regardless of race, color, religion, national origin, sex, age, or physical or mental disability.