Senior ML Infra Engineer for Large-Scale Mid-Training & RL

Peano AI; Palo Alto (CA); On-site; USD 200,000 - 260,000; Full time

Job summary

Peano AI is seeking a Founding Large-Scale Mid-Training/RL Infrastructure Engineer to design and scale our in-house training stack for massive language models. You will work with thousands of GPUs/TPUs, optimize pretraining and RL pipelines, and push the performance envelope for end-to-end training workflows.

The role is deeply technical with ownership across architecture, data pipelines, sharding, and deployment.

Qualifications

• Extensive experience with large-scale DL training and distributed systems.
• Proven record in pretraining or RL-based fine-tuning of large models.
• Hands-on with Megatron, Transformer-Engine, verl, slime or similar toolkits.
• Experience designing data pipelines, sharding, checkpointing, and training ops at scale.

Responsibilities

• Build, optimize, and scale distributed training infrastructure for foundation models.
• Improve throughput, efficiency, scalability, cost, and reliability of pretraining and RL pipelines.
• Architect data loading, sharding, checkpointing, batching, and accelerator use for multi-node training.
• Integrate and optimize RL components: rollout, reward modeling, and environment orchestration.
• Work with Megatron, Transformer-Engine, verl, slime, and related toolkits.
• Tune memory usage, mixed-precision, and runtime performance for massive models.
• Debug and profile training bottlenecks across code, compute, networking, and infra.
• Collaborate with teams to ensure performance and functionality of foundation models.
• Translate research prototypes into production-ready, scalable systems.

Skills

Distributed systems design

Large-scale DL training

Performance tuning

Debugging & profiling

Strong ownership

Tools

Megatron

Transformer-Engine

verl

slime

Job description

Peano AI is seeking a Founding Large-Scale Mid-Training/RL Infrastructure Engineer to design and scale our in-house training stack for massive language models. You will work with thousands of GPUs/TPUs, optimize pretraining and RL pipelines, and push the performance envelope for end-to-end training workflows.

The role is deeply technical with ownership across architecture, data pipelines, sharding, and deployment.