Senior Machine Learning Systems Engineer, Ads ML Experience Platform

EngineersOfAI; United States; Hybrid; USD 100,000 - 130,000; Full time

Job summary

EngineersOfAI is hiring for an experienced engineer familiar with large-scale distributed systems to design and build foundational tooling for next-gen ML development. Responsibilities include developing production-grade orchestration frameworks and building infrastructure for experiment tracking and reproducibility.

The ideal candidate should have 5+ years in platform engineering and experience with workflow and AI systems. Join our mission to accelerate the ML lifecycle!

Qualifications

• 5+ years in infrastructure/platform engineering or large-scale distributed systems.
• 2+ years hands-on experience building ML infrastructure or developer SDKs.
• Experience with distributed data processing systems and workflow technologies.

Responsibilities

• Design and build large-scale offline ML experimentation platforms.
• Develop production-grade training orchestration frameworks.
• Build infrastructure for experiment tracking and model registries.
• Partner with ML engineers to improve experimentation velocity.

Skills

Large-scale distributed systems

ML platforms

Workflow orchestration systems

Distributed data processing (Spark, Flink)

Agentic AI systems

Tools

Kubeflow

Airflow

Ray

Job description

Reddit has a flexible workforce! If you happen to live close to one of our physical office locations our doors are open for you to come into the office as often as you'd like. Don't live near one of our offices? No worries: You can apply to work remotely in any country in which we have a physical presence

Team Overview

We are building the next generation of ML research tools and agentic AI platforms that power machine learning development across Reddit. Our mission is to accelerate the Ads ML lifecycle – from experimentation and training to deployment, evaluation, and autonomous operations – through scalable platform services, intelligent automation, and developer-centric tooling.

Our team owns critical platform capabilities including offline ML experimentation systems, production training orchestration frameworks, ML lifecycle automation and, agentic ML frameworks that enable faster model iterations.

We are looking for an experienced engineer with deep expertise in large-scale distributed systems, ML platforms, and emerging agentic architectures to help define and build the foundational tooling for the next generation of our machine learning devX tooling.

What You’ll Do

• Design and build large-scale offline ML experimentation platforms that enable reproducible research, model development, evaluation, and promotion workflows.
• Develop production-grade training orchestration frameworks supporting distributed training, hyperparameter optimization, model evaluation, and automated retraining.
• Build infrastructure for experiment tracking, metadata management, lineage, artifact versioning, model registries, and reproducibility.
• Partner with ML engineers and researchers to improve experimentation velocity and operational efficiency.
• Build automated workflows for model promotion, rollback, compliance validation, and continuous evaluation.
• Design and build an agentic AI execution platform supporting autonomous and human-in-the-loop workflows, including multi-agent orchestration, memory/context systems, and scalable workflow infrastructure.

What You Bring

• 5+ years in infrastructure/platform engineering or large-scale distributed systems.
• 2+ years of hands‑on experience building and operating production ML infrastructure, developer SDKs, platform APIs, or self‑service AI tooling.
• Experience building workflow orchestration systems, developer platforms, or large-scale automation frameworks.
• Experience with distributed data processing systems such as Spark, Flink, Ray, or equivalent technologies.
• Experience with modern orchestration and workflow technologies such as Kubeflow, Argo, Airflow, or similar frameworks.
• Experience building offline ML experimentation platforms, model registries, experiment tracking systems, or training orchestration frameworks.
• Experience building and operating agentic AI systems, including multi-agent orchestration, autonomous workflows, and agent communication/runtime frameworks (e.g., MCP, A2A, and orchestration systems) is a strong plus
• Experience running end-to-end model development and iteration cycles at scale is a plus