Databricks ETL Developer

Insight Global; Toronto; On-site; CAD 90,000 - 130,000; Full time

Job summary

Insight Global is seeking a Databricks ETL Developer to modernize Capital Markets reporting data assets in Azure and Databricks. You will build scalable data pipelines, develop data warehouse solutions, and support reporting and analytics use cases with strong data engineering expertise.

The role emphasizes Medallion architecture, Data Vault 2.0, and Kimball modeling, with emphasis on data governance and real-time processing capabilities. Collaboration in an Agile environment is expected.

Qualifications

• Hands-on Azure Medallion architecture with ADLS Gen2.
• Strong PySpark and Databricks SQL proficiency.
• Deep Delta Lake knowledge, including ACID and optimization.
• Experience with Unity Catalog and data governance.
• Knowledge of data warehousing concepts (DV2/Kimball).

Responsibilities

• Architect and implement data pipelines following Medallion architecture.
• Ingest data from APIs, files, and Databricks transfers using ADLS Gen2.
• Apply Data Vault 2.0 and Kimball dimensional modeling.
• Implement a Data Quality Framework for integrity and reliability.
• Handle SCD Type 2, late-arriving data, backfill, and restatement.
• Build ETL control/audit frameworks for lineage and monitoring.
• Develop and optimize Delta Lake tables with ACID/OPTIMIZE.
• Build real-time pipelines with Structured Streaming.
• Orchestrate pipelines using Databricks Workflows/Jobs.

Skills

Medallion architecture

PySpark

Databricks SQL

Delta Lake

Structured Streaming

Databricks Workflows

Spark tuning

Data Vault 2.0

Kimball dimensional modeling

Unity Catalog

Delta Live Tables

Genie Space

Power BI

Jira

Confluence

Education

Bachelor's degree in a related field

Tools

Azure

ADLS Gen2

Databricks

Python

Delta Lake tooling

Power BI

Jira

Confluence

Job description

Job Description

We are hiring a Databricks ETL Developer to support a strategic Capital Markets Client Reporting initiative through the modernization and automation of enterprise data processes. This consultant will build scalable data pipelines, develop reporting-focused data warehouse solutions, and deliver high-quality data assets within an Azure and Databricks environment. The ideal candidate will possess strong data engineering and data warehousing expertise and be comfortable supporting reporting and analytics-focused use cases.

Key Responsibilities

• Architect and implement data pipelines following the Medallion (Bronze/Silver/Gold) architecture.

• Ingest data from diverse sources including APIs, flat files, binary files, and Databricks-to-Databricks transfers using Azure Data Lake Storage Gen2 (ADLS Gen2).

• Apply data modeling techniques including Data Vault 2.0 and Kimball Dimensional Modeling.

• Implement a Data Quality Framework to enforce data integrity and reliability standards.

• Handle complex data scenarios such as SCD Type 2, late-arriving data, carry-forward logic, backfill, reprocess, and restatement.

• Build ETL control and audit frameworks for lineage, monitoring, and traceability.

• Develop and optimize Delta Lake tables leveraging ACID transactions, Z-ordering, and OPTIMIZE for performance.

• Build real-time data pipelines using Structured Streaming for low-latency ingestion and processing.

• Orchestrate and schedule pipeline workflows using Databricks Workflows (Jobs).

• Tune Spark jobs through query optimization, partitioning strategies, and cluster/compute sizing.

• Enforce data security policies including Row-Level Security (RLS), Column-Level Security (CLS), and data masking.

• Manage data governance and access control using Unity Catalog.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to [e-mail] learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

Skills and Requirements

• Hands-on experience with Medallion architecture on Azure (ADLS Gen2).

• Strong proficiency in PySpark and Databricks SQL.

• Deep knowledge of Delta Lake, including ACID transactions, Z-ordering, and incremental processing.

• Expertise in Structured Streaming for real-time pipeline development.

• Proficiency with Databricks Workflows (Jobs) for orchestration and scheduling.

• Experience with Spark Declarative Pipelines (Lakeflow, formerly Delta Live Tables).

• Advanced Spark performance tuning, partitioning, caching, and compute sizing.

• Deep knowledge of Data Vault 2.0 and Kimball Dimensional Modeling.

• Solid understanding of Unity Catalog for data governance and security.

• Expertise in exception handling and data quality enforcement.

• Familiarity with Genie Space for natural language data exploration. - Strong data warehousing experience, including designing, building, and supporting a data warehouse rather than working exclusively on ETL pipelines.

• Experience developing data models and warehouse structures that support downstream reporting and analytics.

• Experience supporting Power BI reporting teams and optimizing data for reporting consumption.

• Experience automating manual data uploads and integrating multiple source systems.

• Capital Markets, Corporate Banking, or broader Financial Services experience.

• Experience working in an Agile delivery environment using Jira and Confluence.