Senior Data Engineer

Clear Prism

Canada

On-site

CAD 120,000 - 170,000

Full time

Just now
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Enterprise ML projects
Hands-on Azure Databricks role

Job summary

Clear Prism is seeking a senior data engineer to design and build scalable data pipelines for ML use cases. You will engineer ML-ready data models, feature pipelines, and training/inference datasets, collaborating with data scientists to translate requirements into production-grade solutions.

The role focuses on PySpark, Azure Databricks, Delta Lake, and Azure cloud services, delivering robust data quality and scalable processing in an enterprise setting.

Qualifications

  • 7+ years of hands-on data engineering, with ML/analytics experience.
  • Expert PySpark, complex transformations and distributed processing.
  • Extensive Azure Databricks and lakehouse architecture experience.
  • Strong data modelling for ML, feature engineering, and training/inference datasets.
  • Proven end-to-end ML data pipelines from ingestion to production.
  • Python/SQL proficiency with advanced queries and performance tuning.

Responsibilities

  • Design, develop and maintain scalable data pipelines for ML and analytics.
  • Build ETL/ELT processes to ingest, transform, and prepare large data volumes.
  • Create ML-ready data models, features, and training/inference datasets.
  • Collaborate with Data Scientists and ML Engineers to translate requirements.
  • Develop robust feature engineering pipelines and ensure data quality.
  • Maintain historical, time-aware datasets and prevent data leakage.
  • Leverage Azure Databricks, Delta Lake and Azure Data Lake Storage for scalable processing.
  • Optimize PySpark workloads and implement reliable monitoring and data quality checks.
  • Develop data orchestration workflows and production data integrations.

Skills

PySpark
Python
SQL
Feature Engineering
ML-ready Data Modelling
Data Pipelines Design
Data Governance

Tools

Azure Databricks
Delta Lake
Azure Data Lake Storage Gen2
Azure Data Factory
Databricks Workflows
MLflow / Azure Machine Learning

Job description

Duration:

12-month contract

Role status:

We are seeking a highly skilled Senior Data Engineer with extensive experience designing and developing scalable data pipelines and data models to support Machine Learning (ML) use cases. The ideal candidate will bring deep technical expertise in PySpark, Azure Databricks, and Azure cloud technologies, with a strong focus on feature engineering, ML-ready data modelling, and building production-grade data pipelines.

Principal tasks and responsibilities include:
  • Design, develop, and maintain scalable data pipelines using PySpark and Azure Databricks to support machine learning and advanced analytics initiatives.
  • Build and optimize complex ETL/ELT processes to ingest, transform, and prepare large volumes of structured and semi-structured data for ML applications.
  • Design and implement data models specifically optimized for machine learning use cases, including feature engineering, training datasets, and inference pipelines.
  • Collaborate closely with Data Scientists and ML Engineers to understand data requirements and translate them into efficient, reusable, and scalable data engineering solutions.
  • Develop robust feature engineering pipelines to support feature extraction, transformation, aggregation, and enrichment for ML models.
  • Design and maintain historical and time-aware datasets, ensuring data consistency, accuracy, and prevention of data leakage.
  • Leverage Azure Databricks, Delta Lake, and Azure Data Lake Storage to build scalable and high-performance data processing solutions.
  • Optimize PySpark workloads, including complex transformations, joins, partitioning, and distributed processing performance.
  • Implement automated data quality checks, validation processes, monitoring, and error-handling mechanisms to ensure reliable ML-ready datasets.
  • Develop and maintain data orchestration workflows, incremental processing pipelines, and production data integrations.
  • Apply data engineering best practices related to performance optimization, data governance, scalability, and maintainability.
  • Support the operationalization of ML data pipelines, ensuring reliable and consistent data availability for model training and production inference.
Our client:

A complex enterprise organization leveraging advanced analytics and machine learning to drive business outcomes.

Qualifications and pre-requisites:
  • 7+ years of hands-on experience in Data Engineering, with demonstrated experience building data pipelines for machine learning and advanced analytics use cases.
  • Expert-level proficiency in PySpark, including complex data transformations, distributed processing, optimization, and large-scale data engineering.
  • Extensive hands-on experience with Azure Databricks, including Apache Spark, Delta Lake, Databricks Workflows, and lakehouse architecture.
  • Strong expertise in data modelling for machine learning, including designing ML-ready datasets, feature engineering, historical data structures, and training/inference data preparation.
  • Proven experience designing and implementing end-to-end data pipelines specifically supporting ML model development and production use cases.
  • Strong proficiency in Python and SQL, including complex queries, data transformations, and performance tuning.
  • Hands-on experience with Azure data services, particularly Azure Data Lake Storage Gen2, Azure Data Factory, and Azure Databricks.
  • Strong understanding of feature engineering concepts, including feature extraction, aggregation, transformation, point-in-time accuracy, and prevention of data leakage.
  • Experience with Delta Lake, incremental data processing, and modern lakehouse architecture.
  • Solid understanding of data modelling principles, including dimensional modelling, relational modelling, and analytical data structures.
  • Experience implementing data quality frameworks, automated testing, monitoring, and pipeline orchestration.
  • Demonstrated ability to collaborate with Data Scientists and Machine Learning Engineers to support complex ML initiatives.
  • Familiarity with Databricks Feature Engineering, Feature Store, MLflow, or Azure Machine Learning is considered an asset.
  • Experience with Azure DevOps, CI/CD pipelines, and infrastructure automation is an asset.
  • Strong analytical, problem-solving, and communication skills with the ability to independently deliver complex data engineering solutions.
Additional information or perks:
  • Opportunity to work on complex, enterprise-scale machine learning and data engineering initiatives.
  • Hands-on technical role focused on Azure Databricks, PySpark, and ML data engineering.

Clear Prism does not use artificial intelligence throughout the hiring process.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer - ML-Ready Pipelines, Azure Databricks
Senior Data Engineer - ML-Ready Pipelines, Azure Databricks

Clear Prism • Canada

On-site
CAD 120,000 - 170,000
Enterprise ML projects
Hands-on Azure Databricks role
Sr. Data Engineer
Sr. Data Engineer

Recruiting in Motion • Toronto

Hybrid
CAD 120,000 - 140,000
Senior Data Engineer
Senior Data Engineer

Cpus Engineering Staffing Solutions Inc. • Pickering

Hybrid
CAD 90,000 - 120,000
Data Engineer (Databricks)
Data Engineer (Databricks)

Cpus Engineering Staffing Solutions Inc. • Oshawa

Hybrid
CAD 124,000 - 131,000
Senior Data Engineer
Senior Data Engineer

Michael Page CA • Winnipeg

On-site
CAD 90,000 - 120,000
Databricks Engineer - Azure Data Pipelines & Lakehouse
Databricks Engineer - Azure Data Pipelines & Lakehouse

Compunnel Inc. • Mississauga

On-site
CAD 120,000 - 150,000
Azure Databricks Engineer
Azure Databricks Engineer

Data Elephant • Calgary

On-site
CAD 120,000 - 180,000
Azure Cloud Data Engineer
Azure Cloud Data Engineer

Compunnel Inc. • Mississauga

On-site
CAD 100,000 - 130,000
Senior Python PySpark Data Engineer – Databricks
Senior Python PySpark Data Engineer – Databricks

Compunnel Inc. • Mississauga

Remote
CAD 90,000 - 120,000
Databricks Engineer
Databricks Engineer

Compunnel Inc. • Mississauga

On-site
CAD 120,000 - 150,000