Evaluation Researcher: Real-Time AI Quality & Benchmarks

Acceler8; San Francisco (CA); On-site; USD 140,000 - 190,000; Full time

Job summary

Acceler8 Talent is seeking an experienced Evaluation Researcher to design and build evaluation systems for production speech and language models. You will design evaluation frameworks, build automated pipelines and metrics, and assess conversational AI performance such as latency and turn-taking.

The role requires strong Python and ML research engineering skills, plus experience in production-grade evaluations and benchmarking.

Qualifications

• Strong Python and ML research engineering skills.
• Proven impact on model quality and end-user experience.
• Experience with LLM evaluation, speech evaluation or conversational AI.
• Strong experience building evaluations for production speech or text models.
• Knowledge of modern evaluation methods, benchmarking and experimental design.

Responsibilities

• Design and develop evaluation frameworks for speech and text models.
• Build automated evaluation pipelines, benchmarks and quality metrics.
• Evaluate conversational AI performance, including latency and turn-taking.
• Identify model weaknesses and translate evaluation findings into measurable product improvements.
• Work closely with research and engineering teams to improve production AI systems.

Skills

Python

ML engineering

Evaluation design

Benchmarking

Conversational AI

Job description

Acceler8 Talent is seeking an experienced Evaluation Researcher to design and build evaluation systems for production speech and language models. You will design evaluation frameworks, build automated pipelines and metrics, and assess conversational AI performance such as latency and turn-taking.

The role requires strong Python and ML research engineering skills, plus experience in production-grade evaluations and benchmarking.