Senior ML Engineer — AI Agent Evaluation & Voice Models

Jobot; Austin (TX); On-site; USD 200,000 - 300,000; Full time

Job summary

Aircall is seeking a senior ML engineer to design evaluation engines for AI voice agents, spanning voice and text interfaces. You will train and fine-tune TTS/ASR models and define production-ready metrics and release gates to ensure quality.

You will join a diverse, fast-growing team focused on scalable evaluation pipelines, synthetic data, and benchmarking across languages. The role emphasizes reliability and performance in a busy product environment.

Qualifications

• Bachelor's degree in Computer Science, Machine Learning, Statistics, or a related field.
• 8+ years of overall professional experience, including at least 3 years in ML Engineering or Applied Machine Learning.
• Strong experience evaluating supervised and unsupervised machine learning models, LLMs, and deep learning systems.
• Hands-on experience with failure analysis and LLM evaluation.
• Experience building automated evaluation systems.
• Strong communication skills, with the ability to explain complex technical concepts to both technical and non-technical audiences.
• Practical experience training or fine-tuning voice models (TTS, ASR, or speech-to-speech), including building data pipelines and running experiments.

Responsibilities

• Design and document comprehensive evaluation frameworks for AI agents across voice, chat, and messaging platforms.
• Train and fine-tune voice models (TTS, ASR, and speech-to-speech) using production and synthetic data, iterating on model architecture and data.
• Evaluate AI-generated solutions across training pipelines, experimentation frameworks, debugging processes, and optimization strategies.
• Analyze system design decisions, identifying strengths, weaknesses, and potential failure points.
• Develop annotation guidelines and workflows for human-labeled evaluation data and calibrate LLM-as-judge systems.
• Build and maintain live quality monitoring for deployed AI agents, tracking metrics and detecting data drift.
• Own the metric standards for published AI performance metrics and establish release gates.
• Develop voice-specific evaluation frameworks across accents, languages, noise, interruptions, DTMF inputs, and tool failures.
• Build evaluation foundation across all AI agents products and related tooling to measure accuracy and safety.
• Set up repeatable pipelines for regression testing and benchmarking as models evolve.

Skills

Evaluation frameworks

Voice model training

Automated evaluation systems

Communication

Data pipelines

LLM evaluation

Education

Bachelor's degree in CS/ML/Statistics

Master's or PhD in CS/ML/Statistics

Job description

Aircall is seeking a senior ML engineer to design evaluation engines for AI voice agents, spanning voice and text interfaces. You will train and fine-tune TTS/ASR models and define production-ready metrics and release gates to ensure quality.

You will join a diverse, fast-growing team focused on scalable evaluation pipelines, synthetic data, and benchmarking across languages. The role emphasizes reliability and performance in a busy product environment.