Evaluation Researcher

Acceler8; San Francisco (CA); On-site; USD 140,000 - 190,000; Full time

Job summary

Acceler8 Talent is seeking an experienced Evaluation Researcher to design and build evaluation systems for production speech and language models. You will design evaluation frameworks, build automated pipelines and metrics, and assess conversational AI performance such as latency and turn-taking.

The role requires strong Python and ML research engineering skills, plus experience in production-grade evaluations and benchmarking.

Qualifications

• Strong Python and ML research engineering skills.
• Proven impact on model quality and end-user experience.
• Experience with LLM evaluation, speech evaluation or conversational AI.
• Strong experience building evaluations for production speech or text models.
• Knowledge of modern evaluation methods, benchmarking and experimental design.

Responsibilities

• Design and develop evaluation frameworks for speech and text models.
• Build automated evaluation pipelines, benchmarks and quality metrics.
• Evaluate conversational AI performance, including latency and turn-taking.
• Identify model weaknesses and translate evaluation findings into measurable product improvements.
• Work closely with research and engineering teams to improve production AI systems.

Skills

Python

ML engineering

Evaluation design

Benchmarking

Conversational AI

Job description

I'm partnering with a well-funded frontier AI research startup developing low-latency speech and multimodal AI systems capable of interacting with humans in real time.

They're looking for an experienced Evaluation Researcher to design and build evaluation systems that directly improve the quality of production speech and language models.

The role:

• Design and develop evaluation frameworks for speech and text models
• Build automated evaluation pipelines, benchmarks and quality metrics
• Evaluate conversational AI performance, including response quality, latency and turn-taking
• Identify model weaknesses and translate evaluation findings into measurable product improvements
• Work closely with research and engineering teams to improve production AI systems

What they're looking for:

• Strong experience building evaluations for production speech or text models
• Proven impact on model quality and end-user experience
• Experience with LLM evaluation, speech evaluation or conversational AI
• Strong Python and ML research engineering skills
• Knowledge of modern evaluation methods, benchmarking and experimental design

Why consider it?

• Join a well-backed frontier AI research lab developing next-generation conversational and multimodal AI systems.
• You'll have direct ownership of model evaluation and play a key role in improving how intelligent agents interact with people in real time.

Location:Downtown San Francisco | Fully On-site