Azure Cloud Data Engineer

Compunnel Inc.; Mississauga; On-site; CAD 100,000 - 130,000; Full time

Job summary

Compunnel, Inc. is seeking an Azure Cloud Data Engineer responsible for designing and maintaining scalable cloud-based data platforms. Your role includes developing high-performance ETL pipelines and optimizing large-scale data processing solutions within Azure environments.

The ideal candidate will possess strong Python and SQL skills, along with a solid understanding of cloud data engineering principles and enterprise data management. Candidates with expertise in Databricks and experience in handling large datasets will be preferred.

Qualifications

• 9+ years of overall data engineering experience.
• 5+ years of hands-on Python development experience.
• 5+ years of SQL Server development experience working with large datasets.
• 5+ years of experience developing ETL pipelines using Databricks and PySpark.
• Experience designing and implementing cloud data architectures.

Responsibilities

• Design, develop, and maintain scalable data pipelines.
• Build and deploy Spark applications using Azure Databricks.
• Optimize Databricks jobs for performance.
• Monitor and resolve issues impacting data pipelines.
• Monitor and resolve issues impacting data pipelines.

Skills

Python

SQL

Databricks

ETL pipelines

Data modeling

Azure

Cloud architecture

Data governance

Education

Bachelor's degree in Computer Science or related field

Master's degree in Computer Science or related field

Tools

Azure Synapse

Snowflake

Amazon Redshift

Google BigQuery

Apache Airflow

Job description

The Azure Cloud Data Engineer is responsible for designing, developing, and maintaining scalable cloud-based data platforms and data pipelines within Azure environments. This role focuses on building high-performance ETL pipelines, developing Databricks and PySpark solutions, optimizing large-scale data processing, and implementing enterprise data warehouse architectures. The ideal candidate will have strong expertise in Python, SQL, Azure Databricks, cloud data engineering, and enterprise data platform modernization.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines for data ingestion, transformation, and loading from multiple data sources.
• Build and deploy Spark applications using Azure Databricks and PySpark to process large-scale datasets.
• Optimize Databricks jobs for performance, scalability, and reliability.
• Monitor, troubleshoot, and resolve issues impacting data pipelines and Databricks workloads.
• Design and implement enterprise data warehouse solutions and dimensional data models.
• Develop solutions utilizing Delta Lake, cloud data integration, and modern data architecture principles.
• Develop and maintain applications using Python and SQL.
• Create and optimize complex SQL queries, stored procedures, and reverse engineer existing database processes.
• Design and implement data models supporting OLTP, OLAP, dimensional modeling, facts, and dimensions.
• Process structured, semi-structured, and unstructured data.
• Develop event-driven and streaming data ingestion solutions.
• Implement best practices for data governance, security, quality, and cloud data management.
• Perform code reviews to ensure compliance with development standards and optimal execution patterns.
• Support enterprise cloud data platform migrations and modernization initiatives.
• Collaborate with cross-functional teams to design scalable cloud-based data architectures.

Required Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related technical field.
• 9+ years of overall data engineering experience.
• 5+ years of hands-on Python development experience.
• 5+ years of SQL Server development experience working with large datasets.
• 5+ years of experience developing and deploying ETL pipelines using Databricks and PySpark.
• Experience with Delta Lake, data integration, cloud architecture, and data modeling.
• Experience with enterprise data warehouse design and dimensional modeling.
• Experience processing structured, semi-structured, and unstructured data.
• Experience with event-driven and streaming data processing technologies.Experience with cloud data warehouse platforms such as Azure Synapse, Snowflake, Amazon Redshift, or Google BigQuery.
• Strong understanding of OLTP, OLAP, facts, dimensions, and enterprise data warehousing concepts.
• Experience designing and implementing enterprise cloud data architectures.
• Experience leading enterprise cloud data platform migration initiatives.
• Experience working with banking or financial services clients.
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent written and verbal communication skills.

Preferred Qualifications

• Master's degree in Computer Science, Engineering, or a related technical field.
• Experience with Apache Airflow.
• Experience with cloud-based messaging and analytics platforms.