Azure Data Bricks Consultant

Compunnel, Inc.; Toronto; On-site; CAD 110,000 - 170,000; Full time

Job summary

Compunnel, Inc. in Toronto is seeking an experienced Data Engineer to design, develop, and optimize scalable data solutions using Azure Databricks.

You will build end-to-end ETL/ELT pipelines and Lakehouse architectures, collaborating with cross‑functional teams in a consulting environment to deliver enterprise data solutions. The role requires 7–10 years of experience with Azure Databricks, Delta Lake, and SQL/data modeling, focusing on governance, security, and performance optimization in a

Qualifications

• 7–10 years of data engineering experience.
• Hands-on experience with Azure Databricks and Delta Lake.
• Strong expertise in ETL/ELT pipeline development.
• Experience with Spark Structured Streaming and Auto Loader.
• Strong knowledge of Unity Catalog.
• Proficiency in SQL and data modeling.
• Strong understanding of Data Warehousing concepts.
• Experience with Microsoft Azure cloud services.
• Excellent analytical and problem‑solving skills.

Responsibilities

• Design and develop ETL/ELT pipelines using Azure Databricks.
• Build batch ingestion using Auto Loader and real-time ingestion using Spark Structured Streaming.
• Develop and maintain Delta Lake-based data pipelines.
• Manage Unity Catalog for data governance, security, and access control.
• Create and maintain catalogs, schemas, tables, materialized views, functions, and volumes.
• Develop Slowly Changing Dimensions (SCD Type 1 & Type 2) for dimensional data.
• Build Change Data Capture (CDC) pipelines for incremental data processing.
• Implement Lakehouse Federation and configure foreign catalogs for external data integration.
• Optimize data storage using partitioning and Liquid Clustering.
• Ensure data quality, integrity, and consistency across data platforms.
• Participate in CI/CD implementation and follow DevOps best practices.
• Collaborate with business and technical stakeholders to deliver scalable data solutions.

Skills

Data Engineering

ETL/ELT pipelines

SQL & Data Modeling

Azure Databricks

Delta Lake

Spark Structured Streaming

Unity Catalog

Data Warehousing

Azure Cloud Services

Analytical Skills

Problem Solving

Tools

Azure Databricks

Delta Lake

Spark Structured Streaming

Auto Loader

Unity Catalog

Job description

JOB SUMMARY

Design, develop, and optimize scalable data engineering solutions using Azure Databricks. Build end-to-end ETL/ELT pipelines and Lakehouse architectures. Implement data governance, security, and performance optimization best practices. Collaborate with cross-functional teams in a consulting environment to deliver enterprise data solutions.

Key Responsibilities

• Design and develop ETL/ELT pipelines using Azure Databricks.
• Build batch ingestion using Auto Loader and real-time ingestion using Spark Structured Streaming.
• Develop and maintain Delta Lake-based data pipelines.
• Manage Unity Catalog for data governance, security, and access control.
• Create and maintain catalogs, schemas, tables, materialized views, functions, and volumes.
• Develop Slowly Changing Dimensions (SCD Type 1 & Type 2) for dimensional data.
• Build Change Data Capture (CDC) pipelines for incremental data processing.
• Implement Lakehouse Federation and configure foreign catalogs for external data integration.
• Optimize data storage using partitioning and Liquid Clustering.
• Ensure data quality, integrity, and consistency across data platforms.
• Participate in CI/CD implementation and follow DevOps best practices.
• Collaborate with business and technical stakeholders to deliver scalable data solutions.

Required Qualifications

• 7–10 years of experience in Data Engineering.
• Hands‑on experience with Azure Databricks and Delta Lake.
• Strong expertise in ETL/ELT pipeline development.
• Experience with Spark Structured Streaming and Auto Loader.
• Strong knowledge of Unity Catalog.
• Proficiency in SQL and data modeling.
• Strong understanding of Data Warehousing concepts.
• Experience with Microsoft Azure cloud services.
• Excellent analytical and problem‑solving skills.