Applied Scientist: GenAI & Agent Evaluation for CX

Amazon; Seattle (WA); On-site; USD 143,000 - 193,000; Full time

Benefits offered by this job

Health insurance

RSUs equity

401(k) matching

Job summary

Amazon's CXBT team is seeking an Applied Scientist to shape measurement methodologies for evaluating agent-driven builder experiences on AWS. You will design experiments, reason about architectural decisions, and publish findings that advance agent evaluation and developer experience.

You should have deep GenAI and ML expertise, with NLP model experience and cloud-architecture knowledge. The role requires collaboration with engineers, scientists and product managers in a fast-paced environment.

Qualifications

• 3+ years of building models for business application experience
• PhD, or Master's degree and 4+ years of CS, CE, ML or related field experience
• Experience in patents or publications at top-tier peer-reviewed conferences or journals
• Experience programming in Java, C++, Python or related language
• Experience in any of the following areas: algorithms and data structures, parsing, numerical optimization, data mining, parallel and distributed computing, high-performance computing
• Hands-on experience (academic or industrial) building ML models for natural language processing tasks using GenAI

Responsibilities

• Develop novel scientific methodologies for evaluating autonomous, tool-using agents, advancing the state of the art in agent behavioral measurement and developer-experience research.
• Design rigorous, reproducible experiments that yield causal insight into agent decision-making, controlling for confounds and ensuring findings are grounded in observed behavior rather than self-report.
• Apply deep knowledge of cloud architecture and services to reason about the architectural decisions agents make — the service trade-offs, integration constraints, and deployment considerations that shape what gets built.
• Develop verification and synthesis methods that ensure only evidence-grounded conclusions survive, advancing techniques for reliability and faithfulness in agent evaluation.
• Design evaluation methodologies and metrics that quantify the quality of agent-produced outcomes across dimensions such as correctness, robustness, and security posture.
• Publish and present your work at internal and external scientific venues in the fields of ML.
• Collaborate with scientists, engineers, and product managers to design and implement science-focused systems.
• Proficiency in model development, validation and implementation for NLP applications.
• Excellent verbal and written communication and presentation skills, ability to convey rigorous mathematical concepts and considerations to non-experts.

Skills

GenAI

Machine Learning

NLP

Agent evaluation

Cloud architecture

Education

PhD

Master's + 4+ years CS/CE/ML

Tools

Java

C++

Python

Job description

Amazon's CXBT team is seeking an Applied Scientist to shape measurement methodologies for evaluating agent-driven builder experiences on AWS. You will design experiments, reason about architectural decisions, and publish findings that advance agent evaluation and developer experience.

You should have deep GenAI and ML expertise, with NLP model experience and cloud-architecture knowledge. The role requires collaboration with engineers, scientists and product managers in a fast-paced environment.