Staff ML Engineer: Production AI on SageMaker w/ Trainium

Robots & Pencils; Northern (KY); Hybrid; USD 140,000 - 230,000; Full time

Job summary

Robots & Pencils is seeking an engineer to operate and train models on Amazon SageMaker with AWS Trainium. This role involves digging into hardware‑level details, PyTorch training code, and production SageMaker pipelines to ship real solutions for enterprise clients.

You will work hands-on with PyTorch, SageMaker training/inference, and hardware‑specific compilation, collaborating with client teams to scope and deliver production training workloads rather than experiments.

Qualifications

• Strong, hands‑on PyTorch experience, ideally including distributed or multi‑device training.
• Production experience with Amazon SageMaker for training and/or inference.
• Comfort working close to the hardware layer: you understand device‑specific compilation and can debug issues that are actually about the accelerator, not just the model.
• AWS Trainium or Inferentia (Neurons) experience is a plus.
• Solid Python fundamentals and comfort operating in a client‑facing, production engineering environment.

Responsibilities

• Train and operate models on Amazon SageMaker with AWS Trainium as the underlying compute.
• Write and optimize PyTorch training code with a real understanding of how it compiles and executes on Trainium (NeuronCore architecture, compiler behavior, memory and throughput tradeoffs)
• Diagnose training run issues that show up specifically because of the hardware, not just the model, distinguishing a data or code problem from a compiler or device‑level one
• Translate a request for "a Trainium job" into an actual working, cost‑aware training pipeline, end to end
• Tune distributed training runs for throughput and cost on SageMaker's training infrastructure
• Work directly with client and internal engineering teams to scope and deliver real production training workloads, not experiments that stay in a notebook

Skills

PyTorch

Distributed training

Python

Tools

AWS SageMaker

AWS Trainium

Inferentia

Job description

Robots & Pencils is seeking an engineer to operate and train models on Amazon SageMaker with AWS Trainium. This role involves digging into hardware‑level details, PyTorch training code, and production SageMaker pipelines to ship real solutions for enterprise clients.

You will work hands-on with PyTorch, SageMaker training/inference, and hardware‑specific compilation, collaborating with client teams to scope and deliver production training workloads rather than experiments.