AI Evaluation Scientist — Real-World ML Research

Arena Intelligence, Inc.; San Francisco (CA); On-site; USD 170,000 - 210,000; Full time

Benefits offered by this job

Competitive compensation

Equity

Health benefits

Small, mission-driven team

Impactful projects

Job summary

Arena Intelligence, Inc. in San Francisco seeks machine learning scientists to design and analyze experiments that reveal what makes AI models useful, trustworthy and capable through human preference signals. You will work with engineers, product teams, and researchers to develop methods for comparing models and disentangling performance factors.

This role is deeply interdisciplinary. You’ll contribute to open-ended questions, rigorous evaluation, and research with real-world impact while

Qualifications

• PhD or equivalent research experience in ML, NLP, or Statistics.
• Strong ability to design and analyze experiments with statistical rigor.
• Experience publishing research or contributing to ML/NLP evaluation efforts.

Responsibilities

• Design and conduct experiments to evaluate AI model behavior across reasoning, style, robustness, and user preference dimensions.
• Develop new metrics, methodologies, and evaluation protocols beyond traditional benchmarks.
• Analyze large-scale human voting data to uncover insights into model performance and user preferences.
• Collaborate with engineers to implement research findings into production systems.
• Prototype and test research ideas quickly with rigor and speed.
• Author internal reports and external publications contributing to ML research community.
• Partner with model providers to shape evaluation questions and testing approaches.
• Contribute to scientific integrity and transparency of Arena Intelligence leaderboard and tools.

Skills

Hands-on model training

ML & statistics foundations

Experiment design & analysis

Collaborative mindset

Education

PhD or equivalent

Tools

PyTorch

JAX

TensorFlow

Job description

Arena Intelligence, Inc. in San Francisco seeks machine learning scientists to design and analyze experiments that reveal what makes AI models useful, trustworthy and capable through human preference signals. You will work with engineers, product teams, and researchers to develop methods for comparing models and disentangling performance factors.

This role is deeply interdisciplinary. You’ll contribute to open-ended questions, rigorous evaluation, and research with real-world impact while