Senior Forward Deployed Engineer (Model Customization and Optimization)

Amazon Web Services (AWS); Arlington (VA); On-site; USD 168,000 - 227,000; Full time

Benefits offered by this job

Health insurance

Retirement plan

Parental leave

Mentorship and career growth

Job summary

Amazon Web Services, Inc. is seeking a Senior Forward Deployed Engineer on the Custom Models & Optimization team. You will be the technical lead embedded inside a strategic customer, responsible for building, optimizing, and running custom generative AI models at production scale.

You will shape the technical direction, manage multiple workstreams, and mentor others. Expect up to 25% travel as you push for measurable improvements in quality, latency, and cost, partnering with customer

Qualifications

• Hands-on experience training, fine-tuning, or optimizing deep learning models.
• Experience leading design or architecture of large scale systems.
• Ability to mentor and raise the performance bar within a team.

Responsibilities

• Lead the technical engagement embedded with the customer across multiple workstreams.
• Define the customization strategy to meet quality, latency, and cost targets.
• Architect training and inference systems at scale for LLMs and domain models.
• Lead optimization of model training and inference on AWS Trainium with Neuron SDK.
• Mentor engineers and participate in hiring and code reviews.

Skills

Deep learning

Technical leadership

Model optimization

Education

Bachelor's degree in computer science or equivalent

Tools

SGLang

vLLM

TensorRT-LLM

PyTorch FSDP2

Job description

Description

As a Senior Forward Deployed Engineer (Senior FDE) on the Custom Model and Optimization team, you are the technical lead embedded inside a strategic customer, accountable for building, optimizing, and running the custom generative AI models at the heart of their business. You set the technical direction for model customization and optimization engagements, deciding what to build, how to measure it, and how it reaches production. You combine the technical depth of an AWS senior engineer with the urgency of a startup builder, using your judgment and ownership to deliver measurable business outcomes in quality, latency, and cost.

Senior FDEs work on the hardest problems where models meet systems: building domain foundation models, post-training frontier-scale open-weight models with reinforcement learning, and migrating large training and inference workloads onto AWS Trainium. They take on problems where the business problem itself is ambiguous, define the approach, and break it into workstreams others can own. They make long-term architectural trade-offs, balance customer urgency against sustainable design, and are trusted by customer engineering leadership as a technical peer. They raise the bar for the team through design reviews, code reviews, mentorship, and hiring, and they shape how AWS builds.

Key job responsibilities

• Lead the technical engagement: Embed with the customer as the technical lead for their model customization and optimization program, spanning multiple workstreams. Partner with customer engineering and product leadership to frame the business problem, define success metrics and the evaluation bar, and set technical roadmap that moves from proof of value to production. Communicate trade-offs, risks, and results clearly to executives and engineers alike. Requires up to 25% travel.
• Define the customization strategy: Decide the right path to the customer's quality, latency, and cost targets — prompting, distillation, supervised fine-tuning, preference optimization, reinforcement learning (including RL with verifiable rewards and multi-turn agentic RL), continued pre-training, task-specific adapters, or building a domain foundation model. Own the data strategy, synthetic data generation, and evaluation methodology that make those decisions defensible, and drive them to production for patterns such as coding agents, vertical foundation models, and multi-adapter serving.
• Architect training and inference systems at scale: Design distributed training and RL post-training systems for large dense and mixture-of-experts models using frameworks such as PyTorch FSDP2, Megatron-LM, Miles, and verl, and production inference architectures using engines such as SGLang and vLLM. Architect RL interaction harnesses using frameworks such as NeMo Gym and OpenEnv, defining how agent workflows, tools, environments, and reward functions connect to training and evaluation. Set parallelism, precision, and serving strategies that balance throughput, latency, memory, reliability, and cost per token, and ensure the designs hold up under production load.
• Lead optimization on AWS AI chips: Lead migrations of training and inference workloads to AWS Trainium using the AWS Neuron SDK. Diagnose performance and accuracy-parity gaps against GPU baselines, direct kernel-level optimization where it matters, and work with AWS Neuron and service teams to close capability gaps. Turn benchmark evidence into recommendations that shape customer infrastructure and capacity decisions.
• Lead, mentor, and raise the bar: Provide technical leadership to a team of FDEs and scientists on the engagement, breaking ambiguous problems into well-scoped workstreams and unblocking others. Mentor FDE IIs on model systems design and production ownership. Lead design reviews, provide high-judgment code review, and actively participate in hiring and interviewing.
• Multiply impact across AWS: Identify patterns across engagements and turn them into reusable fine-tuning recipes, evaluation frameworks, benchmarking tools, and reference architectures adopted by other teams. Influence the roadmaps of AWS services, including Amazon Bedrock, Amazon SageMaker AI, and AWS Neuron, with field evidence. Identify and shape opportunities to expand model customization and optimization within your customer and across similar customers.
• Oversee engagement execution: Own the delivery plan, set milestones, track progress against deadlines, and ensure the quality and completeness of all deliverables. Hold yourself and the team accountable for commitments made to the customer, elevate risks early, and adjust scope or resources when timelines are at risk.

About The Team

The Custom Models & Optimization team, part of the AWS Forward Deployed Engineering team, partners with enterprises, AI-native startups, and foundation model providers to build, customize, and optimize generative AI models at scale. Our work spans domain foundation models, post-training of open-weight models, and moving training and inference onto AWS Trainium. As embedded engineers within the AWS Forward Deployed Engineering motion, we bring deep model and optimization expertise to customers' most strategic AI workloads, and we turn what we learn into tooling and patterns that scale across AWS.

About Aws

Diverse Experiences

Amazon values diverse experiences. Even if you do not meet all of the preferred qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying.

Why AWS

Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses.

Work/Life Balance

We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud.

Inclusive Team Culture

AWS values curiosity and connection. Our employee-led and company-sponsored affinity groups promote inclusion and empower our people to take pride in what makes us unique. Our inclusion events foster stronger, more collaborative teams. Our continual innovation is fueled by the bold ideas, fresh perspectives, and passionate voices our teams bring to everything we do.

Mentorship and Career Growth

We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional.

Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status, disability, or other legally protected status.

Los Angeles County applicants: Job duties for this position include: work safely and cooperatively with other employees, supervisors, and staff; adhere to standards of excellence despite stressful conditions; communicate effectively and respectfully with employees, supervisors, and staff to ensure exceptional customer service; and follow all federal, state, and local laws and Company policies. Criminal history may have a direct, adverse, and negative relationship with some of the material job duties of this position. These include the duties and responsibilities listed above, as well as the abilities to adhere to company policies, exercise sound judgment, effectively manage stress and work safely and respectfully with others, exhibit trustworthiness and professionalism, and safeguard business operations and the Company’s reputation. Pursuant to the Los Angeles County Fair Chance Ordinance, we will consider for employment qualified applicants with arrest and conviction records.

Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner.

Basic Qualifications

• 5+ years of non-internship professional software development experience
• 5+ years of programming with at least one software programming language experience
• 5+ years of leading design or architecture (design patterns, reliability and scaling) of new and existing systems experience
• Experience as a mentor, tech lead or leading an engineering team
• Hands-on experience training, fine-tuning, or optimizing deep learning models
• Experience communicating technical concepts to both technical and non-technical audiences, including senior leadership

Preferred Qualifications

• Bachelor's degree in computer science or equivalent
• 2+ years of experience deploying and operating ML models in customer production environments, working with teams that build and scale end-to-end production systems
• Experience defining evaluation strategy and building evaluation frameworks or benchmarks for generative models
• Experience leading optimization of model training or inference for latency, throughput, or cost, using serving engines such as SGLang, vLLM, or TensorRT-LLM, on AWS Trainium (Neuron SDK), on AWS services such as SageMaker AI, SageMaker HyperPod, or EKS, or by writing custom kernels (CUDA, Triton, NKI)
• Experience leading LLM post-training efforts, including SFT, DPO, RLHF, and RL with verifiable rewards or multi-turn agentic RL, using frameworks such as Miles, slime, verl, NeMo-RL, OpenRLHF, SkyRL, or TRL
• Experience leading large-scale distributed training of LLMs, including mixture-of-experts models, on multi-node accelerator clusters, using frameworks such as PyTorch FSDP2, TorchTitan, Megatron-LM/Megatron Core, NeMo, DeepSpeed, or NeuronX Distributed

Amazon also offers comprehensive benefits including health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage), 401(k) matching, paid time off, and parental leave. Learn more about our benefits at https://amazon.jobs/en/benefits.
USA, CA, East Palo Alto - 193,300.00 - 261,500.00 USD annually
USA, MA, Boston - 168,100.00 - 227,400.00 USD annually
USA, NY, New York - 184,900.00 - 250,200.00 USD annually
USA, VA, Arlington - 168,100.00 - 227,400.00 USD annually
USA, WA, Seattle - 168,100.00 - 227,400.00 USD annually

Company

Amazon Web Services, Inc.

Job ID: A10590705