Cloud Engineer

TechDoQuest; Brampton; On-site; CAD 110,000 - 150,000; Full time

Job summary

TechDoQuest in Brampton, Canada is seeking a Senior Azure Data Engineer to design, develop, and optimize scalable data pipelines and cloud data solutions for analytics and reporting. You will lead the development of batch and real-time ETL/ELT workflows using Azure Data Factory, Databricks, and lakehouse architectures, ensuring reliable, well-modelled datasets for Power BI and BI tools.

The role demands strong SQL, Python, PySpark, and DataOps practices across CI/CD pipelines and collaboration

Qualifications

• 6+ years of experience in data engineering with hands-on cloud data platform experience.
• Expert-level SQL and strong data modeling skills.
• Strong proficiency in Python and PySpark.
• Proven experience building and optimizing large-scale ETL/ELT pipelines.
• Hands-on experience with Azure Data Factory, Databricks, and Azure Data Lake.
• Experience with Azure Synapse Analytics or Microsoft Fabric.
• Strong understanding of Apache Spark, distributed processing, and streaming technologies such as Kafka or Azure Event Hubs.
• Experience with Git, Azure DevOps, CI/CD, and DataOps practices.
• Understanding of Power BI data integration requirements, including preparing reporting-ready datasets.

Responsibilities

• Design and maintain scalable ETL/ELT pipelines for batch and real-time data integration.
• Develop optimized SQL queries, data models, and curated datasets for analytics.
• Build data processing solutions using Python, PySpark, and distributed processing frameworks.
• Implement solutions using Azure Data Factory, Databricks, Azure Data Lake, and Synapse or Microsoft Fabric.
• Improve pipeline performance, reliability, data quality, and monitoring.
• Implement DataOps practices and CI/CD pipelines using Azure DevOps.
• Collaborate with analytics teams to provide trusted datasets for Power BI and other BI platforms.

Skills

SQL

Data modeling

Python

PySpark

ETL/ELT pipelines

Cloud data platforms

Azure DevOps / DataOps

Git

Kafka / streaming

Power BI data integration

Tools

Azure Data Factory

Databricks

Azure Data Lake

Azure Synapse Analytics

Microsoft Fabric

Job description

We are looking for a Senior Azure Data Engineer to design, develop, and optimize scalable data pipelines and cloud data solutions. The role focuses on data engineering, integration, and preparing reliable datasets for analytics and reporting.

Key Responsibilities

• Design and maintain scalable ETL/ELT pipelines for batch and real-time data integration.
• Develop optimized SQL queries, data models, and curated datasets for analytics.
• Build data processing solutions using Python, PySpark, and distributed processing frameworks.
• Implement solutions using Azure Data Factory, Databricks, Azure Data Lake, and Synapse or Microsoft Fabric.
• Improve pipeline performance, reliability, data quality, and monitoring.
• Implement DataOps practices and CI/CD pipelines using Azure DevOps.
• Collaborate with analytics teams to provide trusted datasets for Power BI and other BI platforms.

Required Skills

• 6+ years of experience in data engineering, including hands-on cloud data platform experience.
• Expert-level SQL and strong data modeling skills.
• Strong proficiency in Python and PySpark.
• Proven experience building and optimizing large-scale ETL/ELT pipelines.
• Hands-on experience with Azure Data Factory, Databricks, and Azure Data Lake.
• Experience with Azure Synapse Analytics or Microsoft Fabric.
• Strong understanding of Apache Spark, distributed processing, and streaming technologies such as Kafka or Azure Event Hubs.
• Experience with Git, Azure DevOps, CI/CD, and DataOps practices.
• Understanding of Power BI data integration requirements, including preparing reporting-ready datasets. Dashboard development is outside the scope of this role.

Nice to Have

• Experience integrating data with Domo.
• Exposure to Delta Lake, lakehouse architecture, and data governance.