Data Engineer/Developer with Python and SQL

TechDoQuest; Mississauga; On-site; CAD 75,000 - 110,000; Full time

Job summary

TechDoQuest in Mississauga, Canada, seeks a data integration specialist to design, develop, and implement ETL jobs using Python and Spark. You will migrate legacy SSIS/SQL Server pipelines to modern platforms, ensuring data quality and performance.

You will maintain and optimize data workflows for reliability and scale, implement robust testing, and collaborate with data architects, scientists, and analysts in an Agile environment. Strong SQL skills complement your ETL expertise.

Qualifications

• Experience designing and implementing ETL pipelines across platforms.
• Strong Python and Spark proficiency for data processing.
• Advanced SQL skills with complex queries and performance tuning.
• Experience migrating ETL processes from SSIS/SQL Server.
• Experience working in Agile environments.

Responsibilities

• Design, develop, and implement new ETL jobs using Python and/or Spark.
• Migrate ETL processes from SSIS and SQL Server to modern platforms.
• Maintain and optimize data pipelines for efficiency, reliability, and scalability.
• Develop robust testing strategies for ETL jobs to ensure data quality and accuracy.
• Collaborate with data architects, data scientists, and business analysts to translate requirements into technical specs.
• Participate actively in an Agile development environment including stand-ups, sprint planning, and retrospectives.
• Communicate with users and stakeholders to gather requirements, provide updates, and resolve issues.
• Troubleshoot and resolve data-related issues and performance bottlenecks in a timely manner.

Skills

Python

Apache Spark

SQL

ETL development

Data pipelines

Tools

SSIS

SQL Server

Job description

• Design, develop, and implement new ETL (Extract, Transform, Load) jobs using Python and/or Spark to support various data initiatives.
• Migrate existing ETL processes from SSIS and SQL Server to modern data platforms, ensuring data integrity and performance.
• Maintain and optimize existing data pipelines and ETL processes for efficiency, reliability, and scalability.
• Develop and implement robust testing strategies for all ETL jobs to ensure data quality and accuracy.
• Collaborate with data architects, data scientists, and business analysts to understand data requirements and translate them into technical specifications.
• Participate actively in an agile development environment, including stand-ups, sprint planning, and retrospectives.
• Communicate effectively with users, stakeholders, and team members to gather requirements, provide updates, and resolve issues.
• Troubleshoot and resolve data-related issues and performance bottlenecks in a timely manner.
• Strong proficiency in Python and/or Apache Spark for data processing and ETL development.
• Strong SQL knowledge, with proven experience in writing complex queries, stored procedures, and optimizing database performance.