Data Engineer

EPAM Systems; Colombia; Remote; COP 60,000,000 - 90,000,000; Full time

Job summary

EPAM Systems, Inc. seeks a Data Engineer to join a team building an MLOps platform that automates model development, deployment, and feature engineering.

The featureOps component leverages Azure Databricks, Azure Data Factory, and Azure DevOps to build a CI/CD pipeline feeding the Databricks Feature Store for ML model development in Model Lab. Responsibilities include migrating Spark feature pipelines to Azure, collaborating to forecast models, supporting production deployment, and ensuring

Qualifications

• 2+ years of experience working with PySpark.
• Strong background in Data Science core skills.
• Proficiency in feature engineering and ML pipeline development.
• Experience migrating data pipelines to cloud environments.
• English proficiency at B2 level or higher.

Responsibilities

• Migrate existing Spark feature engineering pipelines to Azure.
• Collaborate with the team to identify and develop a forecasting model.
• Support the deployment of the developed machine learning model to production.
• Contribute to the design and development of the CI/CD pipeline feeding the Databricks Feature Store.
• Work closely with Data Scientists and engineers to align feature engineering with ML model requirements.
• Ensure the reliability and scalability of data pipelines across the platform.
• Troubleshoot and optimize existing Spark-based workflows during migration.

Skills

PySpark

Data Science

Feature engineering

ML pipelines

Cloud migration

Tools

MLflow

Docker

Azure Kubernetes Services

Azure Data Factory

Azure Databricks

Azure DevOps

Job description

We are seeking a Data Engineer to join a team developing an MLOps platform that automates machine learning model development, deployment, and feature engineering. The featureOps component of the platform leverages Azure Databricks, Azure Data Factory, and Azure DevOps to build a CI/CD pipeline that feeds the Databricks Feature Store, with generated features used for ML model development in Model Lab.ResponsibilitiesMigrate existing Spark feature engineering pipelines to AzureCollaborate with the team to identify and develop a forecasting modelSupport the deployment of the developed machine learning model to productionContribute to the design and development of the CI/CD pipeline feeding the Databricks Feature StoreWork closely with Data Scientists and engineers to align feature engineering with ML model requirementsEnsure the reliability and scalability of data pipelines across the platformTroubleshoot and optimize existing Spark-based workflows during migrationRequirements2+ years of experience working with PySparkStrong background in Data Science core skillsProficiency in feature engineering and machine learning pipeline developmentExperience migrating data pipelines to cloud environmentsEnglish proficiency at B2 level or higherNice to haveFamiliarity with MLflow and DockerKnowledge of Azure Kubernetes Services, Azure Data Factory, and Azure DatabricksUnderstanding of ML model deployment practicesWe offerInternational projects with top brandsWork with global teams of highly skilled, diverse peersHealthcare benefitsEmployee financial programsPaid time off and sick leaveUpskilling, reskilling and certification coursesUnlimited access to the LinkedIn Learning library and 22,000+ coursesGlobal career opportunitiesVolunteer and community involvement opportunitiesEPAM Employee GroupsAward-winning culture recognized by Glassdoor, Newsweek and LinkedInEPAM is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, age, sexual orientation, gender identity or expression, disability, protected veteran status, or any other characteristic protected by applicable law.