Databricks ETL Consultant

BeachHead Inc.; Toronto; On-site; CAD 120,000 - 180,000; Full time

Job description

Working with one of our top financial clients, this role calls for a Databricks ETL Consultant. You will be instrumental in supporting a critical Capital Markets reporting modernization initiative, building scalable data pipelines within an Azure Databricks environment to facilitate advanced analytics and executive reporting. This hands-on role involves data ingestion, transformation, modelling, and automation across complex data sources, ensuring the delivery of trusted datasets for downstream applications like Power BI.

Responsibilities

• Design, develop, and maintain scalable ETL and data pipelines using Azure Databricks and PySpark.
• Implement Medallion architecture across Bronze, Silver, and Gold data layers for reliable and efficient data processing.
• Ingest and integrate data from diverse sources including APIs, files, SQL databases, and Databricks environments.
• Manage data storage solutions using Azure Data Lake Storage Gen2 and orchestrate workflows with Azure Data Factory.
• Transform raw data into high-quality, trusted datasets optimized for reporting and analytics.
• Apply data warehousing and dimensional modelling principles such as Kimball and Data Vault 2.0.
• Collaborate with Business Systems Analysts (BSA) to translate requirements into effective data structures.
• Implement data quality, validation, exception handling, and audit mechanisms.
• Handle complex data scenarios including slowly changing dimensions, late arriving data, backfills, and reprocessing.
• Develop and optimize Delta Lake tables with incremental processing and performance tuning.
• Automate pipelines and workflows through Databricks Jobs and Azure DevOps pipelines.
• Build real-time or low-latency data pipelines utilizing Structured Streaming where needed.
• Tune Spark workloads via query optimization, partitioning, and caching strategies.
• Enforce data governance and security best practices using Unity Catalog and access controls.
• Support deployment and version control through Azure DevOps and CI/CD pipelines.
• Contribute to the ongoing modernization of the data infrastructure, transitioning from traditional SQL environments to Databricks.

Desired Skill-Set

• Minimum 7-10+ years of experience in data engineering, ETL development, or data warehousing
• Extensive hands-on experience with Azure Databricks
• Proficiency in PySpark and Databricks SQL
• Strong understanding of data warehousing concepts and dimensional modelling
• Experience with Medallion architecture and layered data pipelines
• Deep knowledge of Delta Lake and modern Databricks data engineering practices
• Familiarity with Azure Data Lake Storage Gen2 and Azure Data Factory
• Expertise in Kimball and Data Vault 2.0 modelling techniques
• Design and optimize scalable scheduled or real-time ETL workflows
• Strong SQL and analytical skills
• Experience with data quality controls and ETL governanceWorking knowledge of Azure DevOps and CI/CD processes

Nice to Have

• Experience in Capital Markets or financial services
• Knowledge of Structured Streaming or Delta Live Tables
• Familiarity with Unity Catalog and Databricks Genie
• Supporting enterprise reporting platforms such as Power BI
• Experience working with complex financial datasets like revenue, risk, or profitability
• Working with Lakeflow or Delta Live Tables

BeachHead is an equal opportunity agency and employer. We advocate for our candidates and welcome applicants regardless of race, color, religion, national origin, sex, age, or physical or mental disability. BeachHead or our clients may use technology-enabled tools, including automation and artificial intelligence (AI), to support parts of the recruitment process such as resume screening, application management, and candidate matching. These tools assist our recruiters and our clients, and do not replace human decision-making. This job posting represents a current or anticipated vacancy. The position may be filled at any time, and the posting may be removed without notice once the role has been filled.