Senior AI Systems Engineer: Model Evaluation & QA

Deepgram; United States; Remote; USD 140,000 - 210,000; Full time

Job summary

Deepgram is seeking a Senior Software Engineer - Model Evaluation & AI Systems to join the team responsible for validating the quality of our speech, audio, and multilingual models before they reach customers. You will define evaluation methodologies, build automated pipelines, and collaborate with Research, model training, inference, and product teams to ensure metrics are robust and actionable.

This role emphasizes automation and scalable testing across production stages.

Qualifications

• BS, MS, or PhD in Computer Science, AI, Applied Math, or related field.
• 5+ years of professional software or QA engineering experience.
• Strong backend/scripting experience in Python, Rust, Go, or similar.
• Experience designing and building automated test pipelines, evaluation frameworks, or data-processing systems.
• Strong analytical skills and comfort reasoning about metrics, thresholds, and statistical variation.

Responsibilities

• Define and build evaluation methodologies for Deepgram's models, spanning speech-to-text, text-to-speech, and emerging LLM, RAG, agent, and multimodal systems.
• Design, build, and maintain automated evaluation pipelines across batch and streaming, with a focus on correctness and reproducibility.
• Build scalable evaluation infrastructure — harnesses, orchestration, and result-aggregation pipelines against production models.
• Translate Research benchmarks and expected model metrics into automated, enforceable pass/fail gates.
• Build and operate canaries and continuous-monitoring systems that detect quality regressions in production.
• Partner with DevOps/Infra to stand up ephemeral test environments and results-aggregation infrastructure.

Skills

Python

Rust

Go

QA engineering

Backend development

Data processing

Education

BS/MS/PhD in CS or related field

Tools

Grafana

CI/CD tooling

Job description

Deepgram is seeking a Senior Software Engineer - Model Evaluation & AI Systems to join the team responsible for validating the quality of our speech, audio, and multilingual models before they reach customers. You will define evaluation methodologies, build automated pipelines, and collaborate with Research, model training, inference, and product teams to ensure metrics are robust and actionable.

This role emphasizes automation and scalable testing across production stages.