Databricks Data Engineer

TechDoQuest; Toronto; On-site; CAD 100,000 - 150,000; Full time

Job summary

TechDoQuest in Toronto seeks a seasoned Data Engineer to design scalable ETL/ELT pipelines using Azure Databricks and PySpark, and to migrate BigQuery data to Delta Lake. You will build batch and streaming processes, implement Medallion Architecture, and develop data quality frameworks.

The role requires 7–10 years of cloud data engineering experience, strong data architecture knowledge, and hands-on work with ADLS, Git/CI/CD, and SQL tuning.

Qualifications

• 7–10 years of data engineering experience with cloud-based data solutions.
• Strong understanding of data architecture principles and best practices.
• Bachelor’s degree in computer science or related field.

Responsibilities

• Design and develop scalable ETL/ELT pipelines using Azure Databricks and PySpark.
• Migrate BigQuery datasets, schemas, tables, and views to Delta Lake.
• Build and optimize batch and incremental data processing pipelines.
• Implement Medallion Architecture (Bronze, Silver, Gold) and Delta Lake best practices.
• Develop data validation, reconciliation, and quality-check frameworks.
• Support historical data migration, performance tuning, and production deployment.

Skills

Azure Databricks

PySpark

ETL/ELT Development

Azure Data Lake Storage

Git and CI/CD

SQL Optimization

Data Migration

BigQuery Migration

Education

Bachelor’s degree in Computer Science or related field

Tools

Unity Catalog

Delta Lake

Airflow

DBT

Cloud Composer

Dataflow

BigQuery

Job description

Design and develop scalable ETL/ELT pipelines using Azure Databricks and PySpark.

Migrate BigQuery datasets, schemas, tables, and views to Delta Lake.

Build and optimize batch and incremental data processing pipelines.

Implement Medallion Architecture (Bronze, Silver, Gold) and Delta Lake best practices.

Develop data validation, reconciliation, and quality-check frameworks.

Support historical data migration, performance tuning, and production deployment.

Mandatory Skills

• Proven experience with Azure Databricks, including Unity Catalog and Data Lakehouse Architecture.
• Strong proficiency in PySpark and Spark SQL.
• Expertise in ETL/ELT Development.
• Experience with Azure Data Lake Storage (ADLS).
• Familiarity with Git and CI/CD practices.
• SQL Optimization and Performance Tuning skills.
• Experience in Data Migration and Reconciliation Frameworks.
• GCP BigQuery Migration Experience is essential.

Preferred Skills

• Experience with Airflow / Cloud Composer Dataflow.
• Familiarity with DBT (Data Build Tool) for data transformation.

Qualifications

The ideal candidate will possess a bachelor’s degree in computer science, Information Technology, or a related field. A minimum of 7-10 years of experience in data engineering, with a strong focus on cloud-based data solutions, is required. Candidates should demonstrate a solid understanding of data architecture principles and best practices.