Research Scientist, Agent Evaluation

Reinforced Learning; Town of Tonawanda (NY); Sur place; USD 120 000 - 170 000; Plein temps

Résumé du poste

Reinforced Learning in the United States seeks a role focused on designing evaluations of agents on long-horizon, tool-using tasks that are statistically sound and diagnostic about why a model failed. You will own held-out evaluation suites and publish benchmarks and methodology.

The candidate should have a research background in evaluation or benchmarking, strong statistics, fluent Python, and clear technical writing to provide contamination controls and failure-mode analysis for customers.

Qualifications

• A research background in evaluation, benchmarking or agents.
• Strong statistics and experimental design.
• Fluent Python and experience with evaluation harnesses.
• Clear technical writing.

Responsabilités

• Held-out evaluation suites across our environment library.
• Metrics beyond pass rate: efficiency, recovery from errors, calibration, safety of actions.
• Contamination controls and leakage detection between training and evaluation.
• Failure-mode analysis and the reports customers act on.
• Published benchmarks and methodology.

Connaissances

Research background

Statistics & experimental design

Fluent Python

Technical writing

Python

Outils

Evaluation harnesses

Description du poste

The role

Knowing whether an agent improved is harder than it sounds. You’ll design evaluations of agents on long-horizon, tool-using tasks that are statistically sound, contamination-resistant and diagnostic about why a model failed.

What you’ll own

• Held-out evaluation suites across our environment library.


• Metrics beyond pass rate: efficiency, recovery from errors, calibration, safety of actions.


• Contamination controls and leakage detection between training and evaluation.


• Failure-mode analysis and the reports customers act on.


• Published benchmarks and methodology.

What you bring

• A research background in evaluation, benchmarking or agents.


• Strong statistics and experimental design.


• Fluent Python and experience with evaluation harnesses.


• Clear technical writing.

Helpful, not required

• A public benchmark you released.


• Domain depth in coding, finance or science.