Senior AI Systems Engineer – Model Evaluation

Deepgram; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Deepgram is seeking a Senior Software Engineer - Model Evaluation & AI Systems to join the team responsible for validating the quality of our speech, audio, and multilingual models before they reach customers.

You will define evaluation methodologies, build scalable pipelines, and integrate quality gates into CI/CD, working with Research, training, inference, and product teams to protect the customer experience.

Qualifications

• BS, MS, or PhD in Computer Science, AI, Applied Math, or related field, or equivalent experience.
• 5+ years of professional software or QA engineering experience, with a track record of shipping test infrastructure or evaluation systems.
• Solid backend/scripting experience in Python, Rust, Go, or similar.
• Experience designing and building automated test pipelines, evaluation frameworks, or data-processing systems.
• Strong analytical skills and comfort reasoning about metrics, thresholds, and statistical variation in results.

Responsibilities

• Define evaluation methodologies for models across speech-to-text, TTS, and multimodal systems.
• Build automated evaluation pipelines for batch and streaming with metrics like WER and latency.
• Create scalable evaluation infrastructure and result-aggregation pipelines.
• Translate Research benchmarks into pass/fail gates.
• Build canaries and monitoring systems to detect quality regressions.
• Collaborate with DevOps/Infra to stand up ephemeral test environments.
• Provide trusted evaluation signals to inform release decisions.
• Integrate evaluation into CI/CD to verify quality continuously.

Skills

Backend development

Python

Rust

Go

QA engineering

Distributed systems

Education

BS/MS/PhD in Computer Science/AI

Tools

CI/CD tooling

Testing frameworks

Canaries/monitoring

Grafana

Job description

Deepgram is seeking a Senior Software Engineer - Model Evaluation & AI Systems to join the team responsible for validating the quality of our speech, audio, and multilingual models before they reach customers.

You will define evaluation methodologies, build scalable pipelines, and integrate quality gates into CI/CD, working with Research, training, inference, and product teams to protect the customer experience.