Sr Data Engineer - Mississauga

Randstad Canada; Mississauga; Remote; CAD 120,000 - 160,000; Full time

Benefits offered by this job

Remote work

Databricks/PySpark exposure

Healthcare domain experience

Job summary

Randstad Canada is seeking a Sr Data Engineer to join a platform modernization team on a 6-month remote contract. You will build Delta Lake pipelines, port legacy SQL Server logic to PySpark/Spark SQL, and ensure data quality and traceability across pipelines for a healthcare compliance platform.

The role emphasizes high-volume ingestion, automated testing, and parity reconciliation, operating in an Agile pod with a technical lead, and collaborating across a modern lakehouse architecture.

Qualifications

• 6+ years of data engineering experience, including 3+ years with Databricks and Apache Spark.
• Proficient in Python and PySpark; able to write, test, and tune production-grade scripts.
• Advanced SQL/T-SQL skills with ability to translate complex SQL Server stored procedures.

Responsibilities

• Delta Lake pipeline engineering for high-volume batch and intraday ingestion into Bronze/Silver/Gold layers.
• Migrate legacy SQL Server logic to PySpark and Spark SQL.
• Develop automated data quality tests and reconciliation against legacy outputs.

Skills

Databricks

PySpark

Spark SQL

Python

SQL

T-SQL

Delta Lake

Spark

Tools

Azure Data Factory

Azure Storage

Delta Live Tables

Databricks Workflows

Unity Catalog

Job description

Our client, is seeking an experienced, detail-oriented Sr Data Engineer to join their platform modernization team.

In this technical, hands-on role, you will build pipelines, transformations, and data validation tools to support the modernization of a high-stakes healthcare pharmacy compliance platform using Databricks. Working within an Agile delivery pod under a technical lead, you will engineer high-volume ingestion workloads across a multi-tier Delta Lake architecture (Bronze, Silver, Gold) while porting and re-implementing existing SQL Server stored procedure logic into PySpark and Spark SQL. Because this platform drives financial and regulatory decisions, engineering efforts prioritize data accuracy, automated testing, parity reconciliation against legacy outputs, and full traceability across every pipeline step.

Duration: 6-Month Contract (100% Remote)

Work Arrangement: 100% Remote

Advantages

• 100% Remote Flexibility: Enjoy complete remote work flexibility while contributing to a mission-critical platform modernization initiative.
• Modern Lakehouse Architecture: Hands-on technical execution utilizing Databricks, PySpark, Spark SQL, and Delta Lake design patterns.
• High-Impact Regulatory Domain: Drive data pipelines for a platform handling critical healthcare and pharmacy compliance workflows where precision and data quality are paramount.
• Focused 6-Month Engagement: High-intensity consulting assignment centered on core legacy-to-cloud platform migration and reconciliation.

Responsibilities

• Delta Lake Pipeline Engineering: Build and maintain high-volume batch and intraday ingestion pipelines into a Delta Lake architecture (Bronze, Silver, Gold layers).
• Stored Procedure Migration: Port, translate, and re-implement legacy SQL Server stored procedure business logic into optimized PySpark and Spark SQL code.
• Data Quality & Reconciliation: Develop automated test suites, data quality checks, and output-parity reconciliation jobs comparing new lakehouse results against legacy system outputs.
• Performance Tuning & Optimization: Optimize Spark job execution by tuning partition strategies, file sizing, join types, skew handling, and cluster compute parameters.
• Idempotency & Late Data Handling: Construct idempotent, re-runnable data pipelines designed to gracefully manage late-arriving or corrected data updates.
• CI/CD & Engineering Standards: Maintain version control in Git, participate in peer code reviews, build CI/CD pipeline automation, and document technical design specs.

Qualifications

• Data Engineering Depth: 6+ years of dedicated professional experience in data engineering, including 3+ years of hands-on production experience with Databricks and Apache Spark.
• Python & PySpark Proficiency: Strong skills independently writing, testing, and tuning production-grade Python and PySpark scripts.
• SQL & T-SQL Expertise: Advanced proficiency in SQL and T-SQL, with the demonstrated ability to interpret, dissect, and translate complex SQL Server stored procedures.
• Delta Lake Expertise: Practical experience with Delta Lake capabilities, including MERGE/upsert operations, incremental loads, schema evolution, time travel, and table maintenance.
• Spark Execution Knowledge: Deep understanding of the Spark execution model (jobs, stages, shuffles, join strategies, memory allocation) and performance troubleshooting.
• DevOps & Testing: Proficiency using Git, automated data testing frameworks, and CI/CD tools for data pipeline delivery.
• Communication Skills: Strong written communication skills with a focus on clear technical documentation and team collaboration.

Preferred Assets & Nice-to-Haves

• Experience with healthcare, pharmacy, payer, or life-sciences data (e.g., HL7, ePrescribe, NCPDP, claims, NDC drug data, or HIPAA-regulated PHI).
• Direct experience with SQL Server to Databricks / Lakehouse migrations.
• Familiarity with Azure Data Factory, Azure Storage, Delta Live Tables, Databricks Workflows, and Unity Catalog.
• Exposure to data quality tools such as Great Expectations or transformation frameworks like dbt.
• Active Databricks Certified Data Engineer (Associate or Professional) designation.

Summary

If you are a tech-savvy Sr Data Engineer who pairs 6+ years of enterprise experience with 3+ years of production Databricks/PySpark mastery, T-SQL stored procedure translation skills, and a commitment to data accuracy and testing, this 100% remote 6-month contract opportunity is an ideal match. Bring your Lakehouse engineering precision, Spark tuning skills, and migration expertise to our team today!

Randstad Canada is committed to fostering a workforce reflective of all peoples of Canada. As a result, we are committed to developing and implementing strategies to increase the equity, diversity and inclusion within the workplace by examining our internal policies, practices, and systems throughout the entire lifecycle of our workforce, including its recruitment, retention and advancement for all employees. In addition to our deep commitment to respecting human rights, we are dedicated to positive actions to affect change to ensure everyone has full participation in the workforce free from any barriers, systemic or otherwise, especially equity-seeking groups who are usually underrepresented in Canada’s workforce, including those who identify as women or non-binary/gender non-conforming; Indigenous or Aboriginal Peoples; persons with disabilities (visible or invisible) and; members of visible minorities, racialized groups and the LGBTQ2+ community.

Randstad Canada is committed to creating and maintaining an inclusive and accessible workplace for all its candidates and employees by supporting their accessibility and accommodation needs throughout the employment lifecycle.

This posting is for existing and upcoming vacancies.