Lead Data Engineer

EPAM Systems; México; On-site; MXN 900,000 - 1,500,000; Full time

Job summary

EPAM Systems in Mexico is seeking a Lead Data Engineer to build an MLOps platform, transitioning Spark pipelines to Azure, and collaborating with Data Scientists to forecast models and deploy them to production.

You will design CI/CD pipelines feeding the Databricks Feature Store and work with Azure Data Factory, Databricks, Docker, and Kubernetes to ensure scalable, reliable data pipelines across the platform.

Qualifications

• 5+ years of PySpark experience.
• Proven track record building feature engineering and ML pipelines.
• Experience moving data pipelines to cloud environments.

Responsibilities

• Migrate Spark feature engineering pipelines to Azure Databricks.
• Collaborate with Data Scientists to align features with model requirements.
• Design and implement CI/CD pipelines feeding Databricks Feature Store.
• Deploy ML models to production and monitor performance.
• Ensure reliability and scalability of data pipelines across the platform.
• Troubleshoot and optimize Spark workflows during migration.

Skills

PySpark

Data pipelines

ML deployment

CI/CD for ML

Azure cloud

Databricks

Docker

Kubernetes

Tools

Azure Databricks

Azure Data Factory

Databricks Feature Store

Azure DevOps

MLflow

Docker

Azure Kubernetes Service

Job description

We're looking for a Lead Data Engineer to construct an MLOps platform that supports automated model delivery, deployment, and large-scale feature engineering. In this role, you'll advance the featureOps stack by leveraging Azure Databricks, Azure Data Factory, and Azure DevOps to implement CI/CD pipelines feeding into the Databricks Feature Store for Model Lab applications.ResponsibilitiesTransition existing Spark feature engineering pipelines over to AzureWork alongside the team to design and develop a forecasting modelSupport the deployment of the machine learning model into a production environmentContribute to designing and building the CI/CD pipeline that supplies the Databricks Feature StoreWork in close coordination with Data Scientists and engineers to align feature engineering with ML model requirementsEnsure data pipelines remain reliable and scalable across the entire platformTroubleshoot and enhance current Spark-based workflows during the migration processRequirementsA minimum of 5 years of practical PySpark experienceStrong foundational knowledge in core data science principlesDemonstrated success in building feature engineering and machine learning pipelinesHands-on background in moving data pipelines into cloud-based environmentsB2 (Upper-Intermediate) English proficiency or aboveNice to haveFamiliarity with MLflow and DockerWorking knowledge of Azure Kubernetes Services, Azure Data Factory, and Azure DatabricksUnderstanding of best practices for deploying ML modelsWe offerInternational projects with top brandsWork with global teams of highly skilled, diverse peersHealthcare benefitsEmployee financial programsPaid time off and sick leaveUpskilling, reskilling and certification coursesUnlimited access to the LinkedIn Learning library and 22,000+ coursesGlobal career opportunitiesVolunteer and community involvement opportunitiesEPAM Employee GroupsAward-winning culture recognized by Glassdoor, Newsweek and LinkedInEPAM is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, age, sexual orientation, gender identity or expression, disability, protected veteran status, or any other characteristic protected by applicable law.