Senior Data Engineer - ML-Ready Pipelines, Azure Databricks

Clear Prism

Canada

On-site

CAD 120,000 - 170,000

Full time

Just now
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Enterprise ML projects
Hands-on Azure Databricks role

Job summary

Clear Prism is seeking a senior data engineer to design and build scalable data pipelines for ML use cases. You will engineer ML-ready data models, feature pipelines, and training/inference datasets, collaborating with data scientists to translate requirements into production-grade solutions.

The role focuses on PySpark, Azure Databricks, Delta Lake, and Azure cloud services, delivering robust data quality and scalable processing in an enterprise setting.

Qualifications

  • 7+ years of hands-on data engineering, with ML/analytics experience.
  • Expert PySpark, complex transformations and distributed processing.
  • Extensive Azure Databricks and lakehouse architecture experience.
  • Strong data modelling for ML, feature engineering, and training/inference datasets.
  • Proven end-to-end ML data pipelines from ingestion to production.
  • Python/SQL proficiency with advanced queries and performance tuning.

Responsibilities

  • Design, develop and maintain scalable data pipelines for ML and analytics.
  • Build ETL/ELT processes to ingest, transform, and prepare large data volumes.
  • Create ML-ready data models, features, and training/inference datasets.
  • Collaborate with Data Scientists and ML Engineers to translate requirements.
  • Develop robust feature engineering pipelines and ensure data quality.
  • Maintain historical, time-aware datasets and prevent data leakage.
  • Leverage Azure Databricks, Delta Lake and Azure Data Lake Storage for scalable processing.
  • Optimize PySpark workloads and implement reliable monitoring and data quality checks.
  • Develop data orchestration workflows and production data integrations.

Skills

PySpark
Python
SQL
Feature Engineering
ML-ready Data Modelling
Data Pipelines Design
Data Governance

Tools

Azure Databricks
Delta Lake
Azure Data Lake Storage Gen2
Azure Data Factory
Databricks Workflows
MLflow / Azure Machine Learning

Job description

Clear Prism is seeking a senior data engineer to design and build scalable data pipelines for ML use cases. You will engineer ML-ready data models, feature pipelines, and training/inference datasets, collaborating with data scientists to translate requirements into production-grade solutions.

The role focuses on PySpark, Azure Databricks, Delta Lake, and Azure cloud services, delivering robust data quality and scalable processing in an enterprise setting.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer
Senior Data Engineer

Clear Prism • Canada

On-site
CAD 120,000 - 170,000
Enterprise ML projects
Hands-on Azure Databricks role
Databricks Engineer - Azure Data Pipelines & Lakehouse
Databricks Engineer - Azure Data Pipelines & Lakehouse

Compunnel Inc. • Mississauga

On-site
CAD 120,000 - 150,000
Databricks Engineer
Databricks Engineer

Compunnel Inc. • Mississauga

On-site
CAD 120,000 - 150,000
Senior Data Engineer — AI Data Platform & Pipelines
Senior Data Engineer — AI Data Platform & Pipelines

Quandri Technologies, Inc. • Vancouver

Hybrid
CAD 140,000 - 170,000
Employee stock options
Comprehensive health benefits
4 weeks paid vacation
+2
Azure Cloud Data Engineer
Azure Cloud Data Engineer

Compunnel Inc. • Mississauga

On-site
CAD 100,000 - 130,000
Remote Azure Data Architect: Cloud Data Pipelines
Remote Azure Data Architect: Cloud Data Pipelines

Source Code • Toronto

Hybrid
CAD 120,000 - 160,000
Databricks Architect
Databricks Architect

Acunor • Canada

On-site
CAD 100,000 - 130,000
Senior Data Engineer: Lead Scalable Data Pipelines
Senior Data Engineer: Lead Scalable Data Pipelines

MPA Recruitment • Toronto

On-site
CAD 120,000 - 180,000
Senior Data Engineer - ML Platform
Senior Data Engineer - ML Platform

hireVouch • Canada

Remote
CAD 140,000 - 190,000
Azure Data Platform Engineer – Databricks & Delta Lake
Azure Data Platform Engineer – Databricks & Delta Lake

Compunnel Inc. • Mississauga

On-site
CAD 100,000 - 130,000