Lead AI Solution Engineer with Databricks, ML

EPAM Systems; United States; Remote; USD 150,000 - 230,000; Full time

Job summary

EPAM Systems Inc. is seeking a Lead AI Solution Engineer in the United States who combines deep Databricks expertise with ML to architect, launch, and operate production-grade ML systems. The role blends hands-on data engineering and ML engineering to deliver real-time and batch solutions.

You will design streaming pipelines with Spark Streaming, optimize feature engineering, and ensure low-latency inference while maintaining reliability and scalability across production environments.

Qualifications

• 7+ years of experience building, deploying, and operating offline and online feature engineering pipelines.
• Proficiency in Databricks, Spark/PySpark, Python, and SQL.
• Expertise in model training and real-time inferencing pipelines.
• Experience processing large-scale unstructured data.
• Knowledge of Spark Streaming.
• Experience with real-time feature generation and low-latency model inference.
• Proven track record of running production-grade ML systems.
• English proficiency at B2 level or higher.
• Nice to have Exposure to MLflow and established MLOps workflows.
• Hands-on background with AWS.
• Prior work in Customer Analytics, Recommendation Systems, or Retail industries.

Responsibilities

• Design and optimize feature engineering pipelines for both batch and real-time workloads
• Build, deploy, and maintain scalable model training and inference pipelines
• Process structured and unstructured data at scale
• Develop streaming solutions using Spark Streaming
• Enable real-time feature generation and model serving
• Ensure reliability, scalability, and performance of ML solutions in production
• Support batch pipelines for model training and retraining
• Operate and maintain production ML systems to ensure consistent performance

Skills

Databricks

Spark/PySpark

Python

SQL

Real-time feature generation

Low-latency inference

Production ML systems

AWS

Unstructured data processing

MLOps (MLflow)

Spark Streaming

Customer Analytics / Retail domain

Tools

MLflow

AWS

Spark Streaming

Spark

Job description

We are looking for a Lead AI Solution Engineer who brings strong expertise in Databricks and Machine Learning to architect, launch, and manage production-level ML systems. The ideal candidate works hands-on as a Data Engineer or Machine Learning Engineer, has demonstrated success delivering both real-time and batch ML solutions, possesses solid skills in building streaming pipelines, and brings experience maintaining the scalability and reliability of ML systems once they are live in production.

Responsibilities

• Design and optimize feature engineering pipelines for both batch and real-time workloads
• Build, deploy, and maintain scalable model training and inference pipelines
• Process structured and unstructured data at scale
• Develop streaming solutions using Spark Streaming
• Enable real-time feature generation and model serving
• Ensure reliability, scalability, and performance of ML solutions in production
• Support batch pipelines for model training and retraining
• Operate and maintain production ML systems to ensure consistent performance

Requirements

• 7+ years of experience building, deploying, and operating offline and online feature engineering pipelines
• Proficiency in Databricks, Spark/PySpark, Python, and SQL
• Expertise in model training and real-time inferencing pipelines
• Experience processing large-scale unstructured data
• Knowledge of Spark Streaming
• Experience with real-time feature generation and low-latency model inference
• Proven track record of running production-grade ML systems
• English proficiency at B2 level or higher
• Nice to have Exposure to MLflow and established MLOps workflows
• Hands-on background with AWS
• Prior work in Customer Analytics, Recommendation Systems, or Retail industries