MLOps Data Engineer: Cloud Pipelines & Feature Store

EPAM Systems; Colombia; Remoto; COP 60.000.000 - 90.000.000; Tempo pieno

Descrizione del lavoro

EPAM Systems, Inc. seeks a Data Engineer to join a team building an MLOps platform that automates model development, deployment, and feature engineering.

The featureOps component leverages Azure Databricks, Azure Data Factory, and Azure DevOps to build a CI/CD pipeline feeding the Databricks Feature Store for ML model development in Model Lab. Responsibilities include migrating Spark feature pipelines to Azure, collaborating to forecast models, supporting production deployment, and ensuring

Competenze

• 2+ years of experience working with PySpark.
• Strong background in Data Science core skills.
• Proficiency in feature engineering and ML pipeline development.
• Experience migrating data pipelines to cloud environments.
• English proficiency at B2 level or higher.

Mansioni

• Migrate existing Spark feature engineering pipelines to Azure.
• Collaborate with the team to identify and develop a forecasting model.
• Support the deployment of the developed machine learning model to production.
• Contribute to the design and development of the CI/CD pipeline feeding the Databricks Feature Store.
• Work closely with Data Scientists and engineers to align feature engineering with ML model requirements.
• Ensure the reliability and scalability of data pipelines across the platform.
• Troubleshoot and optimize existing Spark-based workflows during migration.

Conoscenze

PySpark

Data Science

Feature engineering

ML pipelines

Cloud migration

Strumenti

MLflow

Docker

Azure Kubernetes Services

Azure Data Factory

Azure Databricks

Azure DevOps

Descrizione del lavoro

EPAM Systems, Inc. seeks a Data Engineer to join a team building an MLOps platform that automates model development, deployment, and feature engineering.

The featureOps component leverages Azure Databricks, Azure Data Factory, and Azure DevOps to build a CI/CD pipeline feeding the Databricks Feature Store for ML model development in Model Lab. Responsibilities include migrating Spark feature pipelines to Azure, collaborating to forecast models, supporting production deployment, and ensuring