Applied Scientist, Fauna

Amazon; New York (NY); On-site; USD 100,000 - 145,000; Full time

Job summary

Amazon is seeking an Applied Scientist in New York to lead the development of evaluation frameworks and data collection protocols for robotic capabilities. In this cross-functional role, you will focus on measuring, stress-testing, and improving robot behaviors across real-world tasks.

The ideal candidate should be able to design and implement evaluation policies, develop task definitions, and analyze data to enhance system performance. You will work at the intersection of robotics and machine learning, playing a critical role in shaping high-quality datasets and scalable data collection methods.

Qualifications

• Strong experimental mindset with a systems-oriented approach.
• Ability to work across software, robotics, and data pipelines.
• Comfortable turning ambiguous goals into measurable evaluation systems.

Responsibilities

• Design and implement evaluation frameworks for robots.
• Develop success criteria and benchmarking methodologies.
• Create high-quality data collection protocols.
• Analyze evaluation results to identify performance gaps.
• Collaborate with engineering teams for integration.
• Stay updated on advances in robotics and evaluation.

Skills

Robotics

Machine Learning

Data Analysis

Evaluation Methodologies

Teleoperation

Data Pipelines

Job description

Description

We are seeking an Applied Scientist to lead the development of evaluation frameworks and data collection protocols for robotic capabilities. In this role, you will focus on designing how we measure, stress-test, and improve robot behavior across a wide range of real-world tasks. Your work will play a critical role in shaping how policies are validated and how high-quality datasets are generated to accelerate system performance.

You will operate at the intersection of robotics, machine learning, and human-in-the-loop systems, building the infrastructure and methodologies that connect teleoperation, evaluation, and learning. This includes developing evaluation policies, defining task structures, and contributing to operator-facing interfaces that enable scalable and reliable data collection.

The ideal candidate is highly experimental, systems-oriented, and comfortable working across software, robotics, and data pipelines, with a strong focus on turning ambiguous capability goals into measurable and actionable evaluation systems.

Key job responsibilities

• Design and implement evaluation frameworks to measure robot capabilities across structured tasks, edge cases, and real-world scenarios
• Develop task definitions, success criteria, and benchmarking methodologies that enable consistent and reproducible evaluation of policies
• Create and refine data collection protocols that generate high-quality, task-relevant datasets aligned with model development needs
• Build and iterate on teleoperation workflows and operator interfaces to support efficient, reliable, and scalable data collection
• Analyze evaluation results and collected data to identify performance gaps, failure modes, and opportunities for targeted data collection
• Collaborate with engineering teams to integrate evaluation tooling, logging systems, and data pipelines into the broader robotics stack
• Stay current with advances in robotics, evaluation methodologies, and human-in-the-loop learning to continuously improve internal approaches
• Lead technical projects from conception through production deployment
• Mentor junior scientists and engineers