Research Engineer – Distributed Training & RL Environment Infrastructure

MaxIT Consulting - Max Corporate Group

San Francisco (CA)

Hybrid

USD 140,000 - 210,000

Full time

6 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Founding equity
Health, dental and vision coverage
Flexible arrangements for Bay Area and

Job summary

MaxIT Consulting – Max Corporate Group is hiring a Research Engineer to design and operate the training and evaluation infrastructure for a growing AI company in San Francisco. You will own end-to-end systems for long-running agent workflows and large-scale experiments in a hybrid work environment.

You will help build checkpointing, environment orchestration, and simulation environments used for training and evaluation, collaborating with research and deployed teams to scale a robust

Qualifications

  • Strong experience with large-scale distributed systems and containers.
  • Experience building ML infrastructure and training environments.
  • Exposure to checkpointing, orchestration, and multi-GPU clustering.
  • Publications in top venues (e.g., ICLR, NeurIPS) is a plus.

Responsibilities

  • Own checkpoint and restore of full machine state across many machines.
  • Develop the environment orchestrator for thousands of environments.
  • Provision cloud accounts for agents with dynamic lifecycle.
  • Create realistic simulated companies and data for agent training.
  • Be the infra foundation used by research and deployment teams.

Skills

Distributed systems
Containers & OS internals
ML infrastructure
Model training
Professional English

Education

Bachelor's degree in CS or related field
Master's degree in CS (nice to have)

Tools

GPU clusters
Checkpointing tooling
Cluster orchestration

Job description

ML Infrastructure Engineer – Distributed Training & Systems

San Francisco, CA | Hybrid | Full-Time | Permanent Employment | Mid-Level | ASAP Start

MaxIT Consulting – Max Corporate Group is looking for a Research Engineer to build the training and environment infrastructure of a fast-growing AI company in San Francisco. The company builds evaluation environments and post-training infrastructure for long-horizon enterprise agents, and uses that stack to train and deploy custom models for very large technology companies.

It is a small, founding-stage team out of top AI labs and technology companies, already profitable and with real enterprise revenue. This role sits under the whole company: the research and deployed teams build directly on what you own.

THE CHALLENGE

As agent workflows get longer, evaluation and training infrastructure become the bottleneck. You will own the systems that let long runs be paused, rewound and continued, and that start, watch and shut down thousands of heterogeneous environments cheaply and reliably.

Within your first 90 days you will own one of the core systems end to end: checkpoint and restore, the environment orchestrator, or the simulated companies used for training and evaluation.

KEY RESPONSIBILITIES
  • Build checkpoint and restore of full machine state, including memory and GPU state, across many machines.
  • Build the orchestrator that starts, monitors and shuts down thousands of environments, from a single container to a multi-GPU cluster.
  • Provision real cloud accounts for agents, with hundreds created and deleted on demand.
  • Build realistic simulated companies with code, people, tickets, threads and documentation for agent training and evaluation.
  • Be the infrastructure foundation that the research and deployed teams depend on.
WHAT WE ARE LOOKING FOR
  • One to five years of full-time experience (at least one year full time).
  • Deep experience in at least one of: distributed systems, containers and operating-system internals, ML infrastructure or model training.
  • Has trained or served models at real scale (distributed training, inference or serving), or built the environment or benchmark behind a published result.
  • A publication at a top venue (such as ICLR or NeurIPS).
  • Has built and owned infrastructure that other people depended on, beyond a single research script.
  • Comfortable choosing the right tools: no specific language or framework is required.
  • Professional English, as the role works daily with technical teams and enterprise customers.
NICE TO HAVE
  • Experience with checkpointing, cluster orchestration or scheduling of many heterogenous jobs.
  • Compiler, inference or serving work on large models.
  • Simulation or benchmark environments (robotics sim, agent gyms, browser or operating-system environments, multi-agent benchmarks).
  • Master's degree from a strong computer science program.
  • Interest in joining an early-stage company for the long-term upside.
POSITION DETAILS

Work arrangement: Hybrid, three to four days a week in the San Francisco office
Schedule: Full-Time, six-day work week
Employment type: Permanent employment
Seniority: Mid-Level (1 to 5 years)
Start date: ASAP
Location: San Francisco, California, United States

Competitive base salary, founding-level equity, a performance bonus tied to customer deployments, and health, dental and vision coverage. Flexible arrangements for candidates outside the Bay Area are reviewed case by case.

WORK AUTHORIZATION

Candidates must be legally authorized to work in the United States. Visa sponsorship for new petitions is not available; H-1B transfers and OPT with sufficient time remaining are considered.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

ML Infrastructure Engineer – Distributed Training & Systems
ML Infrastructure Engineer – Distributed Training & Systems

MaxIT Consulting - Max Corporate Group • San Francisco (CA)

Hybrid
USD 150,000 - 190,000
Research Engineer – Training and Environment Infrastructure
Research Engineer – Training and Environment Infrastructure

MaxIT Consulting - Max Corporate Group • San Francisco (CA)

Hybrid
USD 120,000 - 180,000
Health, dental and vision coverage
Hybrid work arrangement
Research Engineer - AI Training & Environment Infra
Research Engineer - AI Training & Environment Infra

MaxIT Consulting - Max Corporate Group • San Francisco (CA)

Hybrid
USD 120,000 - 180,000
Health, dental and vision coverage
Hybrid work arrangement
Research Engineer – RL Infrastructure & Agent Environments
Research Engineer – RL Infrastructure & Agent Environments

MaxIT Consulting - Max Corporate Group • San Francisco (CA)

On-site
USD 150,000 - 190,000
Research Engineer, ML Infrastructure
Research Engineer, ML Infrastructure

cognition • San Francisco (CA)

On-site
USD 180,000 - 250,000
ML Infrastructure Engineer
ML Infrastructure Engineer

Objective Partners • San Francisco (CA)

On-site
USD 180,000 - 250,000
Full medical, dental, vision coverage
Flexible PTO
Daily catered lunches
+1
Research Engineer, RL Environments and Infrastructure
Research Engineer, RL Environments and Infrastructure

Hyphen Connect • San Francisco (CA)

Hybrid
USD 180,000 - 240,000
ML Infra Engineer: Distributed Training & Environments
ML Infra Engineer: Distributed Training & Environments

MaxIT Consulting - Max Corporate Group • San Francisco (CA)

Hybrid
USD 140,000 - 210,000
Founding equity
Health, dental and vision coverage
Flexible arrangements for Bay Area and
ML Infra Engineer - Distributed Training & Orchestration
ML Infra Engineer - Distributed Training & Orchestration

MaxIT Consulting - Max Corporate Group • San Francisco (CA)

Hybrid
USD 150,000 - 190,000
AI Researcher / Machine Learning Engineer
AI Researcher / Machine Learning Engineer

Vivo HealthStaff • San Juan (PR)

Hybrid
USD 175,000 - 300,000
Health benefits
Flexible PTO