Senior ML Engineer, Distributed Data Frameworks

Adobe; San Jose (CA); On-site; USD 183,000 - 265,000; Full time

Job summary

Adobe is seeking a Senior Machine Learning Engineer for its Applied Science Data Frameworks team to build scalable data infrastructure powering multimodal AI training and inference. You will design distributed loaders, feature enrichment pipelines, and registry systems for petabyte-scale datasets.

You’ll work with PyTorch-based training workloads, batch inference pipelines, and vector search components on large GPU clusters, while improving CI/CD and observability for robust ML systems.

Qualifications

• 5-6 years of professional experience building and operating distributed systems or data infrastructure in production environments.
• Familiarity with cloud platforms (AWS or Azure) and data platforms such as Databricks or Spark.
• Proficiency in Python and strong software engineering fundamentals — system design, data structures, algorithms.
• Familiarity with ML frameworks such as PyTorch or TensorFlow; hands-on ML experience is a plus but not required.
• Basic familiarity with MLOps practices including CI/CD pipelines, containerization (Docker), and deployment automation.
• Bachelor's degree in Computer Science, Engineering, or a related field; MS is a plus.
• Strong communication skills and ability to collaborate across engineering and research teams.

Responsibilities

• Contribute to building and maintaining distributed training data loaders for multi-source data ingestion and real-time transformations.
• Help implement feature enrichment pipelines and dataset registry systems supporting multimodal model training.
• Build batch inference pipelines for large-scale feature extraction across distributed GPU clusters.
• Develop data processing systems using Spark, Ray, or similar tools for SQL-based data ingestion and Arrow-based storage.
• Support semantic search capabilities and vector databases for dataset discovery and embedding-based retrieval.
• Contribute to CI/CD for ML systems including self-hosted runners, Docker image builds, and deployment automation.
• Collaborate with ML research teams to translate training requirements into scalable data loading infrastructure.
• Write reusable framework components, SDKs, and documentation to accelerate platform adoption.
• Optimize data pipeline performance for latency, throughput, memory, and GPU utilization.
• Contribute to observability and reliability standards for 24/7 training workloads.

Skills

Distributed systems

Python

ML frameworks (PyTorch)

Cloud platforms (AWS/Azure)

Data pipelines

SQL/Arrow

Education

Bachelor’s degree in CS/Engineering

MS degree a plus

Tools

Apache Spark

Ray

Dask

Databricks

OpenSearch

LanceDB

Apache Arrow

Docker

Job description

Adobe is seeking a Senior Machine Learning Engineer for its Applied Science Data Frameworks team to build scalable data infrastructure powering multimodal AI training and inference. You will design distributed loaders, feature enrichment pipelines, and registry systems for petabyte-scale datasets.

You’ll work with PyTorch-based training workloads, batch inference pipelines, and vector search components on large GPU clusters, while improving CI/CD and observability for robust ML systems.