Software Development Manager, SageMaker Training (AWS)

Amazon Inc.; Santa Clara (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Amazon Inc. in Santa Clara, CA, is seeking a Software Development Manager to lead the SageMaker Training team that builds services for fine-tuning and post-training models at scale. You will hire and grow engineers, set technical direction with senior engineers and PMs, and own the roadmap and production service.

You'll work with PyTorch and frameworks like Verl, FSDP, Megatron, and vLLM, advocating for reliability, cost efficiency, and performance across large accelerator fleets.

Responsibilities

• Own the team's charter and roadmap, and defend its tradeoffs with senior leadership.
• Hire, develop and retain engineers in a specialized field with a scarce talent pool.
• Set technical direction with your senior engineers, applied scientists and product managers.
• Decide what the team builds, and what it does not.
• Turn new post-training techniques into supported customer capabilities on a predictable schedule.
• Own the service in production, including on-call health, availability and operational metrics.
• Improve training throughput and cost per run across large accelerator fleets.

Job description

Software Development Manager, SageMaker Training

Interested in building the distributed systems that let customers customize foundation models at scale? The SageMaker Training and Model Customization team builds the services customers use to fine-tune and post-train models against their own accuracy, reliability, and cost targets. We are hiring a Software Development Manager to lead the team that owns those capabilities.

Model customization is the fastest-moving layer of the machine learning stack. Post-training has moved from supervised fine-tuning to reinforcement learning against verifiable rewards, and from single-turn tasks to agentic training where a model learns by acting in an environment over long trajectories. Each shift changes the shape of the workload. Reinforcement learning puts an inference engine inside the training loop. Agentic training adds environments and tool calls, and moves the bottleneck from one run to the next. Your team turns each new technique into a capability customers can use, without rebuilding the platform every time the research moves.

Underneath the techniques, this stays a hard distributed systems problem. Training jobs run across large accelerator fleets for days at a time, and a single node failure can stall the job. The team works in PyTorch and across frameworks including Verl, FSDP, Megatron and vLLM, and contributes upstream to the open source projects the platform depends on.

As the manager you own the team and its charter. You will hire and grow engineers, set technical direction with your senior engineers and product managers, and decide what the team does and does not build. You own the roadmap and defend its tradeoffs with leadership. You own the service in production, including on-call health, operational metrics, and the requests that come with sitting in the critical path of customer training workloads.

Key job responsibilities

• Own the team's charter and roadmap, and defend its tradeoffs with senior leadership.


• Hire, develop and retain engineers in a specialized field with a scarce talent pool.


• Set technical direction with your senior engineers, applied scientists and product managers.


• Decide what the team builds, and what it does not.


• Turn new post-training techniques into supported customer capabilities on a predictable schedule.


• Own the service in production, including on-call health, availability and operational metrics.


• Improve training throughput and cost per run across large accelerator fleets.

A day in the life

Your morning starts in a design review. The team is working out how to keep an inference engine and a training engine in the same loop for reinforcement learning without leaving accelerators idle, and the answer will shape the future of the platform. You deep dive into possible scenarios, because at this scale a node dies routinely and recovery has to be normal rather than exceptional.

Then a charter call. A new post-training technique is three months out of the research literature and two customers are already asking for it. With your senior engineers and product manager you decide whether it becomes a first-class capability, a recipe on top of what you already have, or a no. That decision is yours to make, and the industry does not wait