Staff Research Scientist, AI Benchmarking & Evaluation

Thomas Talent Network, LLC; San Francisco (CA); On-site; USD 140,000 - 210,000; Full time

Job summary

Our client, an AI healthcare software startup, seeks a Member of Technical Staff (Research Scientist) to develop benchmarks and evaluation methodologies for large language models. You will shape how enterprise AI systems are tested before deployment, collaborating with engineering and top AI labs.

Qualifications include 0-3 years in AI/ML research, strong Python, and experience with PyTorch/TF. Familiarity with Django/Flask is a plus; MBA not required.

Qualifications

• 0-3 years of AI/ML research experience focusing on benchmarking and evaluation.
• Experience in team environments, including development sprints, Git best practices, PR reviews.
• Valued: previous AI startup or research lab experience; founder/early employee background.
• Publications in NLP or benchmarking valued but not required; applied work prioritized.
• Experience building new benchmarks and evaluation methodologies preferred.
• Academic background in CS/ML; Masters/PhD preferred; Bachelor's + 0-3 years considered.
• Background from top tech/AI programs preferred.
• NLP research experience with publications preferred.
• Strong Python programming skills for clean, maintainable code.
• Experience with PyTorch/TF and diffusion models and language modeling.
• Interest in LLM infrastructure.
• Experience with Django, Flask, or other Python-based HTTP servers is a plus.
• Strong communication skills; able to give input and accept feedback.

Responsibilities

• Evaluate new AI models as they're released.
• Create fresh benchmarks, hire labelers, construct datasets, and write white papers.
• Improve auto-evaluation methods for generated text.
• Collaborate with engineering to implement and scale evaluation methodologies.
• Work with top AI labs and enterprise customers to understand evaluation needs.

Skills

Python

PyTorch

TensorFlow

NLP research

Benchmarking

Communication

Team collaboration

Research publication

Education

Master's or PhD preferred

Bachelor's degree

Tools

Git

GitHub

Django

Flask

Job description

Our client, an AI healthcare software startup, seeks a Member of Technical Staff (Research Scientist) to develop benchmarks and evaluation methodologies for large language models. You will shape how enterprise AI systems are tested before deployment, collaborating with engineering and top AI labs.

Qualifications include 0-3 years in AI/ML research, strong Python, and experience with PyTorch/TF. Familiarity with Django/Flask is a plus; MBA not required.