Data Engineer Pyspark and Azure

Compunnel Inc.; Southwestern Ontario; On-site; CAD 80,000 - 120,000; Full time

Job summary

A leading technology company is seeking a highly skilled Data Engineer to lead improvements in their Investment Reporting System. The ideal candidate should have experience with Azure services, PySpark, and knowledge of ETL development. Responsibilities include leading the engineering team and implementing cloud-native solutions. Previous experience in the Insurance domain is a must.

Qualifications

• Proven experience as a Lead Engineer or similar role.
• Hands-on expertise in Azure services and PySpark.
• Strong experience in ETL development and database migrations.
• Excellent communication and leadership skills.
• Must have experience in the Insurance domain.

Responsibilities

• Lead the development of ETL pipelines and cloud-native architectures using Azure services.
• Work hands-on with Azure Data Factory, Databricks, Functions, Event Grid, and Service Bus.
• Collaborate with Solution Architects to translate designs into actionable tasks.
• Implement DevOps practices, CI/CD pipelines, and infrastructure automation.
• Provide technical mentorship and problem-solving support to the team.

Skills

Azure services

PySpark

ETL development

Leadership skills

Communication skills

Job description

We are seeking a highly skilled Data Engineer with expertise in PySpark and Azure to lead the improvement of our Investment Reporting System.

This role involves guiding the engineering team, implementing cloud-native solutions, and ensuring adherence to best practices.

Key Responsibilities

• Lead the development of ETL pipelines and cloud-native architectures using Azure services.
• Work hands-on with Azure Data Factory, Databricks, Functions, Event Grid, and Service Bus.
• Collaborate with Solution Architects to translate designs into actionable tasks.
• Implement DevOps practices, CI/CD pipelines, and infrastructure automation.
• Provide technical mentorship and problem-solving support to the team.

Required Qualifications

• Proven experience as a Lead Engineer or similar role.
• Hands‑on expertise in Azure services and PySpark.
• Strong experience in ETL development and database migrations.
• Excellent communication and leadership skills.
• Must have experience in the Insurance domain.