Lead Data Engineer - MLOps & Azure Platform

EPAM Systems; México; Remote; MXN 900,000 - 1,500,000; Full time

Job summary

EPAM Systems in Mexico is seeking a Lead Data Engineer to build an MLOps platform, transitioning Spark pipelines to Azure, and collaborating with Data Scientists to forecast models and deploy them to production.

You will design CI/CD pipelines feeding the Databricks Feature Store and work with Azure Data Factory, Databricks, Docker, and Kubernetes to ensure scalable, reliable data pipelines across the platform.

Qualifications

• 5+ years of PySpark experience.
• Proven track record building feature engineering and ML pipelines.
• Experience moving data pipelines to cloud environments.

Responsibilities

• Migrate Spark feature engineering pipelines to Azure Databricks.
• Collaborate with Data Scientists to align features with model requirements.
• Design and implement CI/CD pipelines feeding Databricks Feature Store.
• Deploy ML models to production and monitor performance.
• Ensure reliability and scalability of data pipelines across the platform.
• Troubleshoot and optimize Spark workflows during migration.

Skills

PySpark

Data pipelines

ML deployment

CI/CD for ML

Azure cloud

Databricks

Docker

Kubernetes

Tools

Azure Databricks

Azure Data Factory

Databricks Feature Store

Azure DevOps

MLflow

Docker

Azure Kubernetes Service

Job description

EPAM Systems in Mexico is seeking a Lead Data Engineer to build an MLOps platform, transitioning Spark pipelines to Azure, and collaborating with Data Scientists to forecast models and deploy them to production.

You will design CI/CD pipelines feeding the Databricks Feature Store and work with Azure Data Factory, Databricks, Docker, and Kubernetes to ensure scalable, reliable data pipelines across the platform.