Senior ML Engineer, Distributed Data Frameworks

Adobe Inc.; San Jose (CA); On-site; USD 152,000 - 265,000; Full time

Job summary

Adobe Inc. is seeking a Senior Machine Learning Engineer to join the Applied Science Data Frameworks team, building foundational infrastructure for large-scale multimodal AI training and inference.

You will work on distributed data loading, feature enrichment pipelines, and dataset management across petabyte-scale GPU clusters, translating research needs into scalable data loading and preprocessing systems.

Qualifications

• 5-6 years of professional experience building and operating distributed systems or data infrastructure in production environments.
• Strong understanding of distributed computing concepts and experience with frameworks like Apache Spark, Ray, Dask, or equivalent.
• Familiarity with cloud platforms (AWS or Azure) and data platforms such as Databricks or Spark.
• Proficiency in Python and strong software engineering fundamentals - system design, data structures, algorithms.
• Familiarity with ML frameworks such as PyTorch or TensorFlow; hands-on ML experience is a plus but not required.
• Basic familiarity with MLOps practices including CI/CD pipelines, containerization (Docker), and deployment automation.
• Bachelor's degree in Computer Science, Engineering, or a related field; MS is a plus.
• Strong communication skills and ability to collaborate across engineering and research teams.

Responsibilities

• Build and maintain distributed training data loaders for large-scale model training.
• Develop feature enrichment pipelines and dataset registry systems for multimodal training.
• Create batch inference pipelines for large-scale feature extraction.
• Process assets through distributed GPU clusters with queue management and fault tolerance.
• Develop data processing systems using Ray, Spark, DuckDB, or similar tools for SQL-based data ingestion and Arrow-based storage.
• Support semantic search capabilities and vector databases (OpenSearch, LanceDB) for dataset discovery.
• Contribute to CI/CD infrastructure, including self-hosted runners, Docker image builds, testing pipelines, and deployment automation.
• Collaborate with ML researchers to translate training requirements into scalable data loading and preprocessing infrastructure.
• Write reusable framework components, SDKs, and documentation to accelerate platform adoption.
• Optimize data pipelines for startup latency, throughput, memory footprint, and GPU utilization.
• Contribute to observability and reliability standards for production data systems supporting 24/7 training workloads.

Skills

Python

Distributed systems

Data engineering

Machine Learning

PyTorch

TensorFlow

Education

Bachelor's degree in CS/Engineering

MS preferred

Tools

Apache Spark

Ray

DuckDB

OpenSearch

LanceDB

Docker

Apache Arrow

Job description

Adobe Inc. is seeking a Senior Machine Learning Engineer to join the Applied Science Data Frameworks team, building foundational infrastructure for large-scale multimodal AI training and inference.

You will work on distributed data loading, feature enrichment pipelines, and dataset management across petabyte-scale GPU clusters, translating research needs into scalable data loading and preprocessing systems.