Senior ETL Software Developer

Cynet Systems Inc; Toronto; On-site; CAD 90,000 - 120,000; Full time

Job summary

A leading company in data engineering is seeking a highly skilled Software Developer with extensive experience in ETL implementation and data quality assurance. The successful candidate will work on critical data pipelines and analytics infrastructure using Microsoft Azure and Databricks. This role involves collaboration with various teams to design robust frameworks and ensure high-quality data delivery. Candidates should possess strong technical skills and certifications in relevant technologies.

Qualifications

• 5+ years of experience in software development and data engineering.
• Expertise in Microsoft Azure + Databricks and Power BI technology stack.
• Experience in automated testing and CI/CD practices.

Responsibilities

• Design and implement data quality frameworks and automated testing pipelines.
• Collaborate with data engineers to ensure adherence to Medallion Architecture.
• Monitor and assess pipeline performance and data integrity.

Skills

ETL implementation

Data engineering

CI/CD processes

Data quality assessments

Automated testing

Azure DevOps

Power BI

PySpark

SQL

Education

Microsoft Certified: Azure Data Engineer Associate

Databricks Certified Data Engineer Associate/Professional

Microsoft Certified: Fabric Analytics Engineer Associate

Microsoft Certified: Power BI Data Analyst Associate

Microsoft Certified: DevOps Engineer

Tools

JIRA

Azure DevOps

Selenium

JMeter

Visual Studio Code

Job description

Job Description:

Responsibilities:

1. We are seeking a highly skilled Software Developer with ETL implementation experience to join our data engineering team.
2. This role is critical in ensuring the accuracy, reliability, and performance of our data pipelines and analytics infrastructure built on the Microsoft Azure platform and Databricks.
3. Under the supervision of the IT QA Lead, you will collaborate closely with data engineers and DevOps teams to design and implement robust data quality frameworks, automated testing pipelines, and CI/CD processes.
4. Design and implement comprehensive test plans for data pipelines and transformations in a Microsoft Azure + Databricks and Power BI technology stack.
5. Conduct data quality assessments, validation, and profiling using tools like Great Expectations.
6. Develop and maintain automated testing frameworks integrated into CI/CD pipelines.
7. Collaborate with data engineers to ensure adherence to Medallion Architecture best practices.
8. Monitor and assess pipeline performance, scalability, and data integrity.
9. Implement and manage DevOps practices using Azure DevOps, Git, and related tools.
10. Document test cases, validation rules, and QA processes clearly and consistently.
11. Perform functional testing on Power BI Reports and dashboards.
12. Participate in Agile ceremonies and contribute to sprint planning and retrospectives.

Experience and Skill Set Requirements:

Public Sector Experience - 5 points:

1. 5+ years of experience working in the federal/provincial/broader public-sector as a software developer / data specialist / quality assurance tester.
2. 5+ years of experience in data engineering.
3. Must have demonstrated expertise in designing and implementing comprehensive data pipelines and transformations in a Microsoft Azure + Databricks and Power BI technology stack.
4. Experience with Microsoft Azure Data Services (ADF, ADLS Gen 2, Synapse, Azure SQL).
5. Experience with Azure Databricks.
6. Proficiency in PySpark and SQL.
7. Experience with Power BI (data preparation, modelling, and visualization).
8. Proven experience in developing and maintaining data pipelines and release management through CI/CD and DevOps practices using Azure DevOps, Git, Visual Studio Code, and related tools.
9. Experience collaborating with data engineers to ensure adherence to Medallion Architecture (Staging, Bronze, Silver, Gold layers).

Certifications:

• Microsoft Certified: Azure Data Engineer Associate
• Databricks Certified Data Engineer Associate/Professional
• Microsoft Certified: Fabric Analytics Engineer Associate
• Microsoft Certified: Power BI Data Analyst Associate
• Microsoft Certified: DevOps Engineer

Quality Assurance Skill and Experience - 20 points:

1. 5+ years of experience in quality assurance and automated testing.
2. Experience conducting software quality assessments, systems testing, performance evaluation, and automated testing.
3. Proficiency with JIRA or Azure DevOps for defect tracking.
4. Experience with Selenium for automated testing of web-based applications, including Power BI reports and dashboards.
5. Experience with JMeter for performance evaluation.
6. Experience conducting data quality assessments, validation, and profiling using tools like Great Expectations.
7. Experience developing and maintaining automated testing frameworks integrated into CI/CD pipelines and DevOps practices using Azure DevOps, Git, Visual Studio Code, and related tools.
8. Certifications such as Certified Software Tester (CSTE) and Certified Tester Test Automation Engineer are preferred.