ML Infrastructure Engineer: Training, Inference & Kernels

Trajectory; San Francisco (CA); On-site; USD 150,000 - 230,000; Full time

Job summary

Trajectory is seeking an ML Infrastructure Engineer to build the platform that powers training, inference, and kernels for next‑gen AI systems. You will own distributed training, low‑latency serving, and GPU‑kernel optimizations, shaping reusable systems that scale in production.

You will collaborate with researchers to turn new algorithms into reliable components, build benchmarks and observability, and push measurable gains while preserving correctness.

Qualifications

• Strong fundamentals in distributed systems, networking, storage, and failure recovery, with experience shipping and operating demanding systems.
• Deep specialization in training infrastructure, inference systems, or GPU kernels, supported by systems built or measurable optimizations delivered.
• Strong Python skills and languages relevant to your specialty, such as C++, CUDA, or Triton.
• Understanding of PyTorch, JAX, or comparable framework internals, with strong profiling and debugging skills.
• Ownership and clear communication: work closely with researchers and deliver measurable performance gains while preserving correctness and reliability.
• We value demonstrated capability over credentials.

Responsibilities

• Build and optimize distributed training and RL infrastructure, including rollout execution, GPU scheduling, checkpointing, and recovery. Improve training experimentation throughput and shorten research iteration cycles.
• Optimize serving for production agents and training rollouts. Improve batching, scheduling, and KV-cache management while balancing latency, throughput, cost, and model quality.
• Develop and optimize GPU kernels and runtimes using CUDA, Triton, or comparable tools. Improve memory use and execution efficiency, preserving numerical correctness and verifying gains in real workloads.
• Build reproducible benchmarks, observability, and automated research workflows that propose changes, run experiments, and validate improvements. Work with researchers to turn new algorithms into reliable systems across training, inference, and kernels.
• We’re building one of the world’s best continual learning loops for ML infrastructure: agents propose optimizations, run experiments, measure gains, and learn from the results across training, inference, and kernels.

Skills

Distributed systems

Networking

Storage

Failure recovery

GPU kernels

Python

CUDA

C++/CUDA/Triton

Profiling & debugging

Ownership & communication

Tools

CUDA

Triton

PyTorch

Job description

Trajectory is seeking an ML Infrastructure Engineer to build the platform that powers training, inference, and kernels for next‑gen AI systems. You will own distributed training, low‑latency serving, and GPU‑kernel optimizations, shaping reusable systems that scale in production.

You will collaborate with researchers to turn new algorithms into reliable components, build benchmarks and observability, and push measurable gains while preserving correctness.