Lead Data Engineer

EPAM Systems; Colombia; On-site; COP 120,000,000 - 240,000,000; Full time

Benefits offered by this job

Healthcare benefits

Paid time off

Job summary

EPAM Systems is seeking a Lead Data Engineer to build an MLOps platform enabling automated model delivery and large-scale feature engineering. You will migrate Spark pipelines to Azure Databricks, design forecasting models, and deploy ML models into production.

Collaborate with data scientists and engineers to align feature engineering with ML requirements. The role emphasizes building CI/CD pipelines feeding the Databricks Feature Store, ensuring scalable, reliable data pipelines across the

Qualifications

• Minimum 5 years of practical PySpark experience.
• Strong foundation in core data science principles.
• Proven success building feature engineering and ML pipelines.
• Hands-on cloud data pipeline experience.

Responsibilities

• Transition Spark feature engineering pipelines to Azure.
• Work with the team to design and develop forecasting models.
• Support deployment of ML models into production environments.
• Contribute to CI/CD pipelines for Databricks Feature Store.
• Collaborate with Data Scientists and engineers to align features with model needs.

Skills

PySpark

ML pipelines

Feature engineering

CI/CD pipelines

Cloud data platforms

Tools

MLflow

Docker

Azure Databricks

Azure Data Factory

Azure Kubernetes Service

Job description

We're looking for a Lead Data Engineer to construct an MLOps platform that supports automated model delivery, deployment, and large-scale feature engineering. In this role, you'll advance the featureOps stack by leveraging Azure Databricks, Azure Data Factory, and Azure DevOps to implement CI/CD pipelines feeding into the Databricks Feature Store for Model Lab applications.ResponsibilitiesTransition existing Spark feature engineering pipelines over to AzureWork alongside the team to design and develop a forecasting modelSupport the deployment of the machine learning model into a production environmentContribute to designing and building the CI/CD pipeline that supplies the Databricks Feature StoreWork in close coordination with Data Scientists and engineers to align feature engineering with ML model requirementsEnsure data pipelines remain reliable and scalable across the entire platformTroubleshoot and enhance current Spark-based workflows during the migration processRequirementsA minimum of 5 years of practical PySpark experienceStrong foundational knowledge in core data science principlesDemonstrated success in building feature engineering and machine learning pipelinesHands-on background in moving data pipelines into cloud-based environmentsB2 (Upper-Intermediate) English proficiency or aboveNice to haveFamiliarity with MLflow and DockerWorking knowledge of Azure Kubernetes Services, Azure Data Factory, and Azure DatabricksUnderstanding of best practices for deploying ML modelsWe offerInternational projects with top brandsWork with global teams of highly skilled, diverse peersHealthcare benefitsEmployee financial programsPaid time off and sick leaveUpskilling, reskilling and certification coursesUnlimited access to the LinkedIn Learning library and 22,000+ coursesGlobal career opportunitiesVolunteer and community involvement opportunitiesEPAM Employee GroupsAward-winning culture recognized by Glassdoor, Newsweek and LinkedInEPAM is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, age, sexual orientation, gender identity or expression, disability, protected veteran status, or any other characteristic protected by applicable law.