AI Infrastructure Engineer — Multicloud GPU Fleet

mercor.com; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Performance bonus

Equity grant

Relocation bonus

Housing stipend

Meals stipend

Equinox membership

Laundry reimbursement

Wellness reimbursement

Health insurance

Dental insurance

Vision insurance

Job summary

Mercor is seeking an experienced software engineer to own the accelerator platform end-to-end. You will manage GPU clusters across cloud and on‑prem, write production code daily, and design systems that keep accelerators busy and highly utilized.

You will work on scheduling, LLM serving optimization, benchmarking and cost attribution, with on‑call responsibilities. This is a hands-on, production‑oriented role in San Francisco offices.

Qualifications

• 5+ years building and operating production distributed systems or ML infrastructure.
• Proficient Python and Go or C++, with shell scripting.
• Strong Kubernetes and HPC experience with Slurm.
• Experience with distributed training or serving stacks (PyTorch Distributed, Ray, vLLM, Megatron-LM).
• Experience with at least one major cloud provider.
• Familiar with observability tools and on-call duties.

Responsibilities

• Design, deploy and operate GPU clusters across AWS, GCP, and on‑prem data centers.
• Build scheduling to keep accelerators busy: gang scheduling, bin-packing, multi‑tenant quotas.
• Run distributed training infra with PyTorch Distributed, Ray or Megatron.
• Deploy and tune high‑throughput LLM serving with vLLM and quantization.
• Build benchmarking and traffic‑replay tools to measure latency & baselines.
• Health checks and automated remediation across hardware, network and software.
• Instrument fleet utilization with DCGM, Prometheus and Grafana; write alerts.
• Ingest utilization and billing data to support capacity planning.
• Drive autoscaling and idle-capacity recovery with measurable cost results.
• Write runbooks and explain infra tradeoffs to researchers and leadership.

Job description

Mercor is seeking an experienced software engineer to own the accelerator platform end-to-end. You will manage GPU clusters across cloud and on‑prem, write production code daily, and design systems that keep accelerators busy and highly utilized.

You will work on scheduling, LLM serving optimization, benchmarking and cost attribution, with on‑call responsibilities. This is a hands-on, production‑oriented role in San Francisco offices.