Voice AI Evaluation Engineer

Aircall; San Francisco (CA); On-site; USD 181,000 - 250,000; Full time

Benefits offered by this job

Competitive salary package & benefits

Job summary

Aircall is seeking a senior ML Engineer to design evaluation frameworks for its AI agents across voice, chat, and messaging. The role involves training voice models, assessing AI solutions, and building robust evaluation pipelines to ensure accuracy and safety in production.

The position requires hands-on experience with LLMs, failure analysis, and automated evaluation systems, plus strong communication across technical and non-technical teams.

Qualifications

• BS in Computer Science, ML, Statistics, or related field.
• 3+ years in ML Engineering or Applied ML with 8+ years total exp.
• Strong experience evaluating supervised, unsupervised, LLMs.
• Hands-on failure analysis and evaluating LLMs.
• Experience building automated evaluation systems.
• Strong communication across technical and non-technical audiences.
• Hands-on training or fine-tuning voice models (TTS/ASR) including data pipelines.

Responsibilities

• Design and document comprehensive evaluation frameworks for Aircall’s AI agents across voice, chat and messaging.
• Train and fine-tune voice models (TTS, ASR) using production and synthetic data, iterating on architecture and data mix.
• Assess AI generated solutions across training pipelines, experimentation setups, debugging processes, and optimization strategies.
• Analyze system design decisions and identify strengths, weaknesses, and potential failure points.
• Design annotation guidelines and workflows for human-labeled evaluation data; calibrate LLM-as-judge systems.
• Build and maintain live quality monitoring for deployed AI agents, tracking accuracy and safety signals.
• Own the metric contract for every published AI metrics and their validity windows.
• Build release gates and offline regression suites for AI surfaces before changes ship.

Skills

ML Engineering

Applied ML

Evaluation systems

Failure analysis

Voice models

Communication

Education

BS in Computer Science

MS/PhD preferred

Tools

Python

LLM evaluation tools

Job description

Aircall is seeking a senior ML Engineer to design evaluation frameworks for its AI agents across voice, chat, and messaging. The role involves training voice models, assessing AI solutions, and building robust evaluation pipelines to ensure accuracy and safety in production.

The position requires hands-on experience with LLMs, failure analysis, and automated evaluation systems, plus strong communication across technical and non-technical teams.