Machine Learning Engineer

Jobot; Austin (TX); On-site; USD 200,000 - 300,000; Full time

Job summary

Aircall is seeking a senior ML engineer to design evaluation engines for AI voice agents, spanning voice and text interfaces. You will train and fine-tune TTS/ASR models and define production-ready metrics and release gates to ensure quality.

You will join a diverse, fast-growing team focused on scalable evaluation pipelines, synthetic data, and benchmarking across languages. The role emphasizes reliability and performance in a busy product environment.

Qualifications

• Bachelor's degree in Computer Science, Machine Learning, Statistics, or a related field.
• 8+ years of overall professional experience, including at least 3 years in ML Engineering or Applied Machine Learning.
• Strong experience evaluating supervised and unsupervised machine learning models, LLMs, and deep learning systems.
• Hands-on experience with failure analysis and LLM evaluation.
• Experience building automated evaluation systems.
• Strong communication skills, with the ability to explain complex technical concepts to both technical and non-technical audiences.
• Practical experience training or fine-tuning voice models (TTS, ASR, or speech-to-speech), including building data pipelines and running experiments.

Responsibilities

• Design and document comprehensive evaluation frameworks for AI agents across voice, chat, and messaging platforms.
• Train and fine-tune voice models (TTS, ASR, and speech-to-speech) using production and synthetic data, iterating on model architecture and data.
• Evaluate AI-generated solutions across training pipelines, experimentation frameworks, debugging processes, and optimization strategies.
• Analyze system design decisions, identifying strengths, weaknesses, and potential failure points.
• Develop annotation guidelines and workflows for human-labeled evaluation data and calibrate LLM-as-judge systems.
• Build and maintain live quality monitoring for deployed AI agents, tracking metrics and detecting data drift.
• Own the metric standards for published AI performance metrics and establish release gates.
• Develop voice-specific evaluation frameworks across accents, languages, noise, interruptions, DTMF inputs, and tool failures.
• Build evaluation foundation across all AI agents products and related tooling to measure accuracy and safety.
• Set up repeatable pipelines for regression testing and benchmarking as models evolve.

Skills

Evaluation frameworks

Voice model training

Automated evaluation systems

Communication

Data pipelines

LLM evaluation

Education

Bachelor's degree in CS/ML/Statistics

Master's or PhD in CS/ML/Statistics

Job description

Job Details

Build the Evaluation Engine Behind Production AI Voice Agents | $200k-$300k

Salary: $200,000 - $300,000 per year

Base salary range: $181,000 - $250,000 USD

We are a fast‑growing, AI‑powered communications platform transforming how businesses connect with their customers. For over a decade, we've believed voice builds the strongest business relationships. Our platform brings voice, SMS, WhatsApp, and more than 250 integrations together in one seamless workspace, helping businesses drive revenue, resolve customer issues faster, and build lasting relationships. Our AI-powered voice agents handle routine interactions, allowing teams to focus on the conversations that matter most.

Today, our platform is trusted by more than 24,000 businesses worldwide. We’re 600+ people from 40+ nationalities across Paris, London, Madrid, Berlin, New York, San Francisco, Seattle, Sydney, and Mexico City. We’re ambitious, product-driven, and execution-focused—built for visible impact, fast decisions, and real growth.

Why join us?

• Key moment to join Aircall in terms of growth and opportunities
• Our people matter, work-life balance is important at Aircall
• Fast-learning environment, entrepreneurial and strong team spirit
• 45+ Nationalities: cosmopolite & multi-cultural mindset
• Competitive salary package & benefits
• Competitive compensation package and comprehensive benefits

DE&I Statement

At Aircall, we believe diversity, equity and inclusion – irrespective of origins, identity, background and orientations – are core to our journey. We pride ourselves on promoting active inclusion within our business to foster a strong sense of belonging for all. We’re working to create a place filled with diverse people who can enrich and learn from one another. We’re committed to ensuring that everyone not only has a seat at the table but is valued and respected at it by providing equal opportunities to develop and thrive. We will constantly challenge ourselves to make sure that we live up to our ambitions around diversity, equity and inclusion, and keep this conversation open. Above all else, we understand and acknowledge that we have work to do and much to learn.

Job Details

• Design and document comprehensive evaluation frameworks for AI agents across voice, chat, and messaging platforms.
• Train and fine-tune voice models (TTS, ASR, and speech-to-speech) using production and synthetic data. Continuously iterate on model architecture, data composition, and training strategies to improve accuracy, naturalness, and latency.
• Evaluate AI-generated solutions across training pipelines, experimentation frameworks, debugging processes, and optimization strategies.
• Analyze system design decisions, identifying strengths, weaknesses, and potential failure points.
• Develop annotation guidelines and workflows for human-labeled evaluation data. Calibrate LLM-as-judge systems against human reviewers to maintain reliable automated evaluations over time.
• Build and maintain live quality monitoring for deployed AI agents, tracking accuracy, resolution rates, and safety signals in production while identifying model or data drift before it impacts customers.
• Own the metric standards for all published AI performance metrics, including definitions, populations, data granularity, rollups, and validity windows.
• Establish release gates and offline regression testing suites that every AI feature must pass before changes to prompts, models, or configurations are deployed. Measure reliability across repeated trials rather than relying solely on average pass rates.
• Develop voice-specific evaluation frameworks using simulated callers across accents, languages, background noise, interruptions, DTMF inputs, and tool failures, with latency and speech recognition accuracy as key quality metrics.
• Build evaluation foundation across all AI agents products and other agentic products and work on voice models, agent capability evaluations, benchmark design, LLM-as-judge systems, failure analysis, and infrastructure that ties it together by establishing shared metrics, test sets, and tooling to measure accuracy, resolution quality, and safety consistently across products.
• You will set up repeatable pipelines for regression testing and benchmarking as models and features evolve so teams can ship confidently without re-inventing evaluation methodology for each product.

Minimum Qualifications

• Bachelor's degree in Computer Science, Machine Learning, Statistics, or a related field.
• 8+ years of overall professional experience, including at least 3 years in ML Engineering or Applied Machine Learning.
• Strong experience evaluating supervised and unsupervised machine learning models, LLMs, and deep learning systems.
• Hands‑on experience with failure analysis and LLM evaluation.
• Experience building automated evaluation systems.
• Strong communication skills, with the ability to explain complex technical concepts to both technical and non-technical audiences.
• Practical experience training or fine-tuning voice and speech models (TTS, ASR, or speech-to-speech), including building data pipelines and running experiments.

Preferred Qualifications

• Master's or PhD in Computer Science, Machine Learning, Statistics, or a related field.
• Experience evaluating LLMs or agentic systems, including LLM-as-a-judge and RAG evaluation.
• Experience with synthetic data generation and prompt engineering.
• Experience training or fine-tuning voice models at scale, with familiarity in synthetic data generation, model distillation, and low-latency inference optimization for production AI applications.

Equal Opportunity Statement

Jobot is an Equal Opportunity Employer. We provide an inclusive work environment that celebrates diversity and all qualified candidates receive consideration for employment without regard to race, color, sex, sexual orientation, gender identity, religion, national origin, age (40 and over), disability, military status, genetic information or any other basis protected by applicable federal, state, or local laws. Jobot also prohibits harassment of applicants or employees based on any of these protected categories. It is Jobot’s policy to comply with all applicable federal, state and local laws respecting consideration of unemployment status in making hiring decisions.

Sometimes Jobot is required to perform background checks with your authorization. Jobot will consider qualified candidates with criminal histories in a manner consistent with any applicable federal, state, or local law regarding criminal backgrounds, including but not limited to the Los Angeles Fair Chance Initiative for Hiring and the San Francisco Fair Chance Ordinance.