AI Benchmarking & Evaluation Scientist

snorkelai

United States

A distancia

USD 120.000 - 180.000

Jornada completa

Hace 2 días
Sé de los primeros/as/es en solicitar esta vacante

Consigue una respuesta de este empleador

Tu currículum y tu carta de presentación, adaptados exactamente a lo que buscan contratar.

Supera los filtros ATS

Descripción de la vacante

snorkelai is seeking a researcher/engineer to help make evaluations of advanced AI systems more reliable by identifying weaknesses in benchmarks and collaborating with authors to fix them. You will combine hands-on testing of models and agents with research into recurring evaluation failures and engineering improvements that can be reused across tasks.

Requirements include ML/AI evaluation experience, strong Python skills for building and debugging evaluation systems, and the ability to quantify

Formación

  • Experience in ML, LLMs, AI agents, and evaluation.
  • Strong Python skills for building, running, and debugging evaluation systems.
  • Ability to identify limitations in evaluation methods, develop and test alternatives.
  • Experience designing experiments, analyzing model behavior, and investigating failures.
  • Familiarity with benchmarks, evaluation frameworks, or automated graders.
  • Ability to take research or engineering work from discovery through implementation and validation, and communicate findings clearly.

Responsabilidades

  • Probe benchmarks for grading errors, ambiguous tasks, exploitable shortcuts, reward hacking, and environment vulnerabilities; reproduce issues so they can be investigated.
  • Propose repairs for confirmed problems, implement selected changes with benchmark authors, and check that fixes preserve valid solutions.
  • Build or adapt tools and methods that can detect recurring benchmark weaknesses across tasks and datasets.
  • Share findings with authors, coordinate improvements, and help prepare updated benchmark releases.
  • Contribute to technical reports, open-source tools, publications, and new approaches to evaluating benchmarks.

Conocimientos

ML research
Python programming
Evaluation methods
Agent evaluation

Descripción del empleo

snorkelai is seeking a researcher/engineer to help make evaluations of advanced AI systems more reliable by identifying weaknesses in benchmarks and collaborating with authors to fix them. You will combine hands-on testing of models and agents with research into recurring evaluation failures and engineering improvements that can be reused across tasks.

Requirements include ML/AI evaluation experience, strong Python skills for building and debugging evaluation systems, and the ability to quantify

Consigue la evaluación confidencial y gratuita de tu currículum.

o arrastra y suelta tu archivo aquí

Similar jobs

Puestos de trabajo similares que vale la pena comparar

Benchmark Red Team Research Scientist
Benchmark Red Team Research Scientist

Snorkel AI • EE. UU.

Presencial
USD 200.000 - 340.000
Research Fellow: Open Benchmarks & AI Evaluation
Research Fellow: Open Benchmarks & AI Evaluation

Snorkel AI • San Francisco (CA)

Híbrido
USD 117.000 - 227.000
Research advisor
Compute resources
Open-source artifact
+1
Red Team AI Benchmark Scientist & Engineer
Red Team AI Benchmark Scientist & Engineer

Snorkel AI • New York (NY), San Francisco (CA)

Híbrido
USD 200.000 - 340.000
AI Benchmarks Research Fellow
AI Benchmarks Research Fellow

Aitrainergigs • San Francisco (CA), New York (NY)

Híbrido
USD 117.000 - 227.000
Co-advisor
Compute resources
Open-source artifact
+2
Red Team Research Scientist: AI Benchmark Evaluation
Red Team Research Scientist: AI Benchmark Evaluation

Remote Genie • EE. UU.

Presencial
USD 200.000 - 340.000
Lead Open Benchmark Grants & Evaluation Research
Lead Open Benchmark Grants & Evaluation Research

snorkelai • EE. UU.

A distancia
USD 231.000 - 390.000
Senior Software Engineer — Remote AI/ML Systems & Eval
Senior Software Engineer — Remote AI/ML Systems & Eval

JoinStartup • San Francisco (CA)

A distancia
USD 208.000 - 315.000
Frontier AI Benchmark Research Scientist
Frontier AI Benchmark Research Scientist

AItoolnavio • New York (NY), San Francisco (CA)

Híbrido
USD 200.000 - 340.000
Research Fellow, Open AI Benchmarks & Datasets
Research Fellow, Open AI Benchmarks & Datasets

Snorkel AI • EE. UU.

Híbrido
USD 117.000 - 227.000
Research co-advisor
Compute support
Open-source artifact
+2
Research Scientist, Open Benchmarks Red Team
Research Scientist, Open Benchmarks Red Team

Snorkel AI • EE. UU.

Presencial
USD 200.000 - 340.000