MTS – Evals

Seldon Technologies, Inc; Zürich; Hybrid; CHF 120,000 - 180,000; Full time

Benefits offered by this job

Visa sponsorship and relocation

Professional development budget

Health, dental, and vision coverage

Gym membership

Food stipend with near full meal cover

Transportation cost support

Team off-sites and conference attend.

Subscriptions for essential AI tools

Job summary

Seldon Technologies, Inc. is seeking a results-driven evaluator to develop benchmarks for multimodal models and agents, collaborating with domain experts to define tasks and assess outcomes.

You will define success criteria, implement checks for generated files and model responses, and compare automated scores with expert assessments. A strong background in Python and statistics is essential for interpreting results and communicating limitations.

Qualifications

• Strong Python skills for processing datasets, implementing evaluation logic, and analyzing results.
• Understanding of statistics and experimental design, including sampling, uncertainty, and confounding factors.
• Experience evaluating models or testing complex software, and an interest in investigating unexpected results.
• Clear technical writing that explains evaluation methods, findings, and limitations.
• You take initiative and follow through on your work, are principled in decisions, open about uncertainty, and collaborative.

Responsibilities

• Define tasks, success criteria, and evaluation datasets with researchers and domain experts.
• Implement checks for generated files, application state, and model responses, and compare automated scores with expert assessments.
• Run model comparisons, analyze failures, and identify opportunities for improvement and preventing reward hacking.
• Investigate data leakage, ambiguous tasks, and sources of variation in scores; maintain versioned tasks and reproducible runs.

Skills

Python

Statistics

Model evaluation

Technical writing

Initiative & collaboration

Education

PhD preferred

Tools

NumPy

pandas

pytest

Hugging Face Datasets

Job description

You’ll develop benchmarks for multimodal models and agents, working with domain experts to define useful tasks and assess the results. This can involve evaluating an answer about a video or reviewing the steps an agent took in an application . You’ll study whether our tests measure the intended skills and help the team understand what the results mean for further research.

What you’ll work on

• Define tasks, success criteria, and evaluation datasets with researchers and domain experts.
• Implement checks for generated files, application state, and model responses, and compare automated scores with expert assessments.
• Run model comparisons, analyze failures, and identify opportunities for reward hacking. Strengthen tasks and automated checks to prevent models from earning high scores without completing the intended work. Read the traces.
• Investigate data leakage, ambiguous tasks, and sources of variation in scores; maintain versioned tasks and reproducible evaluation runs.

What you’ll bring

• Strong Python skills for processing datasets, implementing evaluation logic, and analyzing results with tools such as NumPy and pandas.
• An understanding of statistics and experimental design, including sampling, uncertainty, and confounding factors.
• Experience evaluating models or testing complex software, and an interest in investigating unexpected results.
• Clear technical writing that explains evaluation methods, findings, and limitations.
• You take initiative and follow through on your work. You’re principled in your decisions, open about uncertainty, and collaborative in how you solve problems.

Nice to have

• Experience with model APIs, Hugging Face Datasets, pytest, or tools for visualizing experimental results.
• Work on benchmark design, reward hacking, data contamination, or human evaluation.
• Familiarity with CAD tools and their scripting APIs, video analysis, or computer-use agents.
• A PhD in a relevant field and research publications are preferred, but neither is required.
• Comprehensive health, dental, and vision coverage.
• Visa sponsorship and relocation support.
• A professional development budget, the best AI tools, GPUs, a Mac, and the subscriptions you need.
• Team off-sites and conference attendance.
• Support with transportation costs.
• Gym membership and sports club subscriptions to support your wellbeing.
• A food stipend with nearly full meal coverage.