Senior AI Systems Engineer — Model Evaluation

Worky; California (MO); On-site; USD 180,000 - 230,000; Full time

Job summary

Deepgram is seeking a Senior Software Engineer for Model Evaluation and AI Systems to join the team responsible for validating the quality of speech, audio, and multilingual models before they reach customers. You will define evaluation methodologies and build scalable pipelines that measure model quality at scale across batch and streaming environments.

This role focuses on automated tests, pass/fail gates, and monitoring that protect the customer experience, collaborating with Research, model

Qualifications

• 5+ years of professional software or QA engineering experience, with a track record of shipping test infrastructure or evaluation systems.
• Strong background in backend development and scripting.
• Ability to design and implement automated test pipelines and evaluation frameworks.

Responsibilities

• Define evaluation methodologies for models across speech-to-text, text-to-speech, and multimodal systems.
• Build automated evaluation pipelines for batch and streaming workloads with metrics like latency and accuracy.
• Create scalable evaluation infrastructure and result-aggregation pipelines.
• Translate benchmarks into automated pass/fail gates and monitor production quality.
• Partner with research, training, and product teams to inform release decisions.
• Integrate evaluation and quality gates into CI/CD workflows.

Skills

Software QA engineering

Analytical thinking

Cross-team communication

Ambiguity handling

Education

BS/MS/PhD in CS/AI/Applied Math

Tools

Python

Rust

Go

Job description

Deepgram is seeking a Senior Software Engineer for Model Evaluation and AI Systems to join the team responsible for validating the quality of speech, audio, and multilingual models before they reach customers. You will define evaluation methodologies and build scalable pipelines that measure model quality at scale across batch and streaming environments.

This role focuses on automated tests, pass/fail gates, and monitoring that protect the customer experience, collaborating with Research, model