Founding ML Infra Engineer — Equity & Open-Source AI

Peano AI; Palo Alto (CA); On-site; USD 180,000 - 260,000; Full time

Job summary

Peano AI is building the Agent Cloud infrastructure and seeks a Founding ML Infrastructure Engineer to design and optimize high‑performance model serving for large‑scale open‑source models in Palo Alto.

You will own end‑to‑end ML systems, work with accelerators like TPUs/GPUs, improve latency and throughput, and collaborate with research and product teams to ship production‑grade infrastructure.

Qualifications

• Strong experience in ML systems, distributed systems, or high-performance computing.
• Experience optimizing inference or training workloads for large models.
• Familiarity with TPUs, GPUs, or other accelerators.
• Experience with CUDA, Triton, NCCL, JAX/XLA, PyTorch internals, vLLM, SGLang, or TensorRT‑LLM.
• Strong systems debugging skills.
• Comfort working across model code, runtime, infrastructure, and product requirements.
• High ownership in an early-stage startup environment.

Responsibilities

• Optimize large-scale LLM inference and serving systems.
• Improve tokens per second, latency, throughput, and cost efficiency.
• Work on serving infrastructure for open-source models across accelerators.
• Improve batching, scheduling, KV cache management, memory usage, and accelerator utilization.
• Support long-context inference up to 1M context.
• Debug performance bottlenecks across model execution, runtime, networking, and infrastructure.
• Work with frameworks such as JAX/XLA, PyTorch, vLLM, SGLang, TensorRT-LLM, or related systems.
• Collaborate with the application team to optimize infrastructure for agentic workloads.
• Help turn research prototypes into reliable production systems.

Skills

ML systems

Distributed systems

High-performance computing

Inference optimization

Training optimization

TPUs

GPUs

Tools

CUDA

Triton

NCCL

JAX/XLA

PyTorch internals

vLLM

SGLang

TensorRT-LLM

Job description

Peano AI is building the Agent Cloud infrastructure and seeks a Founding ML Infrastructure Engineer to design and optimize high‑performance model serving for large‑scale open‑source models in Palo Alto.

You will own end‑to‑end ML systems, work with accelerators like TPUs/GPUs, improve latency and throughput, and collaborate with research and product teams to ship production‑grade infrastructure.