ML Scientist, AI Evaluation & Benchmarking

Arena Intelligence, Inc.; San Francisco (CA); On-site; USD 140,000 - 220,000; Full time

Benefits offered by this job

Competitive compensation

Comprehensive health and wellness benefits

Cutting-edge AI projects

Culture of transparency and trust

Job summary

A leading AI evaluation firm based in San Francisco seeks a Machine Learning Scientist to foster understanding of AI model performance. You'll engage in designing and analyzing comprehensive experiments while collaborating across teams. Applicants should possess a PhD in a relevant field and hands-on experience with large-scale models. The role offers competitive compensation and comprehensive benefits, fostering a culture centered around transparency and community impact.

Qualifications

• Hands-on experience training large-scale models including reward and preference models.
• Strong foundation in ML and statistics for designing training objectives.
• Experience publishing research or working on open-source projects in ML, NLP, or AI.

Responsibilities

• Design and conduct experiments evaluating AI model behavior across various dimensions.
• Develop new metrics and evaluation protocols that exceed traditional benchmarks.
• Collaborate with engineers to implement and scale research findings.

Skills

Experience training large-scale models

Strong foundation in ML and statistics

Proficiency in Python and ML libraries

Demonstrated ability to design experiments

Collaboration with engineering teams

Education

PhD in Machine Learning, Natural Language Processing, or a related field

Tools

Python

PyTorch

JAX

TensorFlow

Job description

A leading AI evaluation firm based in San Francisco seeks a Machine Learning Scientist to foster understanding of AI model performance. You'll engage in designing and analyzing comprehensive experiments while collaborating across teams. Applicants should possess a PhD in a relevant field and hands-on experience with large-scale models. The role offers competitive compensation and comprehensive benefits, fostering a culture centered around transparency and community impact.