Senior AI Model Evaluation & Systems Engineer

Doist; United States; Remote; USD 140,000 - 210,000; Full time

Job summary

Deepgram is hiring a Senior Software Engineer - Model Evaluation & AI Systems to join the team that validates the quality of speech, audio, and multilingual models before customers see results.

You will design evaluation methodologies, build pipelines, and ensure rigorous QA across production models, including LLM and multimodal systems. Strong scripting and backend skills are essential.

Qualifications

• 5+ years of professional software or QA engineering experience.
• Experience shipping test infrastructure or evaluation systems.
• Strong analytical skills and comfort with metrics and thresholds.

Responsibilities

• Define evaluation methodologies for models across speech, text, and multimodal systems.
• Design and maintain automated evaluation pipelines for batch and streaming workflows.
• Build scalable evaluation infrastructure and result aggregation pipelines.
• Translate benchmarks into automated pass/fail gates and monitor quality in production.
• Integrate evaluation gates into CI/CD to verify quality continuously.

Skills

QA engineering

Test infrastructure

Data analysis

Communication

Education

BS/MS/PhD in CS/AI/Math

Tools

Python

Rust

Go

Job description

Deepgram is hiring a Senior Software Engineer - Model Evaluation & AI Systems to join the team that validates the quality of speech, audio, and multilingual models before customers see results.

You will design evaluation methodologies, build pipelines, and ensure rigorous QA across production models, including LLM and multimodal systems. Strong scripting and backend skills are essential.