Senior Speech LLM Evaluator & Benchmark Architect

Apple, Inc.; Cambridge (MA); On-site; USD 166,000 - 292,000; Full time

Benefits offered by this job

Stock programs

RSU awards

Employee stock plan discount

Medical & dental coverage

Retirement benefits

Discounted products

Job summary

Apple, Inc. Cambridge, MA is seeking a Sr. Machine Learning Engineer to own data and metrics for evaluating speech LLMs, building datasets and automated judges for robust, real-time evaluation across multilingual scenarios.

You will partner with modeling, infra, and product teams to ensure rigorous evaluation before deployment, contributing to personal-context understanding and privacy-first design within Siri.

Qualifications

• Bachelor's degree in CS/EE or related field, or equivalent practical experience.
• Experience building or working with text, speech or audio evaluation pipelines and metrics.
• Proficiency in Python and building data processing pipelines at scale.
• Experience curating or annotating datasets for machine learning evaluation.
• Working knowledge of statistics for measuring model performance and interpreting results.
• Familiarity with LLM evaluation techniques, automated and human evaluation.
• Strong written and verbal communication to explain evaluation results.

Responsibilities

• Designs and curates audio evaluation datasets reflecting real-world usage.
• Defines and implements evaluation metrics for audio LLMs across accuracy and quality.
• Builds automated evaluation pipelines and LLM-as-judge tooling.
• Analyzes evaluation results to identify gaps and improvements.
• Collaborates with human-evaluation programs to align automated metrics with judgments.
• Works with infrastructure teams to integrate new datasets and metrics.
• Contributes evaluation methodologies for emerging audio LLM capabilities.

Skills

Python

Data pipelines

Dataset curation

Statistics

LLM evaluation

Communication

Multilingual evaluation

Audio evaluation

Education

Bachelor's degree in CS/EE or related field

Tools

Spark

Job description

Apple, Inc. Cambridge, MA is seeking a Sr. Machine Learning Engineer to own data and metrics for evaluating speech LLMs, building datasets and automated judges for robust, real-time evaluation across multilingual scenarios.

You will partner with modeling, infra, and product teams to ensure rigorous evaluation before deployment, contributing to personal-context understanding and privacy-first design within Siri.