Lead Data Engineer: Architect MLOps on Azure Databricks

EPAM Systems; Colombia; Remote; COP 120,000,000 - 240,000,000; Full time

Benefits offered by this job

Healthcare benefits

Paid time off

Job summary

EPAM Systems is seeking a Lead Data Engineer to build an MLOps platform enabling automated model delivery and large-scale feature engineering. You will migrate Spark pipelines to Azure Databricks, design forecasting models, and deploy ML models into production.

Collaborate with data scientists and engineers to align feature engineering with ML requirements. The role emphasizes building CI/CD pipelines feeding the Databricks Feature Store, ensuring scalable, reliable data pipelines across the

Qualifications

• Minimum 5 years of practical PySpark experience.
• Strong foundation in core data science principles.
• Proven success building feature engineering and ML pipelines.
• Hands-on cloud data pipeline experience.

Responsibilities

• Transition Spark feature engineering pipelines to Azure.
• Work with the team to design and develop forecasting models.
• Support deployment of ML models into production environments.
• Contribute to CI/CD pipelines for Databricks Feature Store.
• Collaborate with Data Scientists and engineers to align features with model needs.

Skills

PySpark

ML pipelines

Feature engineering

CI/CD pipelines

Cloud data platforms

Tools

MLflow

Docker

Azure Databricks

Azure Data Factory

Azure Kubernetes Service

Job description

EPAM Systems is seeking a Lead Data Engineer to build an MLOps platform enabling automated model delivery and large-scale feature engineering. You will migrate Spark pipelines to Azure Databricks, design forecasting models, and deploy ML models into production.

Collaborate with data scientists and engineers to align feature engineering with ML requirements. The role emphasizes building CI/CD pipelines feeding the Databricks Feature Store, ensuring scalable, reliable data pipelines across the