Software Test Engineer

Deepgram; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

Deepgram, a leading voice AI platform in the United States, seeks a Software Test Engineer to design and maintain automated test frameworks across products, models, APIs, and data platforms. You will develop regression tests, build robust test suites, and ensure high-quality releases through close collaboration with QA, Research, Product, Data, and Engineering teams.

You will translate product requirements into test strategies, create representative test data, and validate model-powered features

Qualifications

• BS, MS, or PhD in Computer Science, AI, Applied Math, or related field, or equivalent experience.
• 5+ years of professional software or QA engineering experience, with a track record of shipping test infrastructure or evaluation systems.
• Solid backend/scripting experience in Python, Rust, Go, or similar.
• Experience designing and building automated test pipelines, evaluation frameworks, or data-processing systems.
• Strong analytical skills and comfort reasoning about metrics, thresholds, and statistical variation in results.
• Ability to take charge of ambiguous technical challenges and communicate effectively across teams.

Responsibilities

• Define and execute well-designed test plans across products, APIs, SDKs, model-powered features, and data platforms to ensure robustness and performance.
• Design, build, and maintain automated test suites and frameworks for functional, integration, end-to-end, regression, API, browser, and service-level testing across batch and streaming workflows.
• Translate requirements into clear test strategies, repeatable test cases, and enforceable release gates.
• Build and maintain representative, customer-focused, and adversarial test datasets and environments.
• Validate model-powered behavior (speech-to-text, text-to-speech, and AI features) using metrics, human review, and regression coverage.
• Build testing infrastructure, including harnesses, scripts, data tooling, dashboards, and release-readiness reporting.
• Integrate automated tests, quality checks, canaries, and release validation into CI/CD pipelines.
• Collaborate with Engineering, Product, Research, Data, Infrastructure, and DevOps to understand system behavior and deployment risks.
• Test data ingestion, processing, annotation, and quality-control workflows for data integrity and representativeness.
• Execute staging and production validation, load testing, cross-browser and customer-workflow testing, and user acceptance testing with stakeholders.
• Maintain the test-case repository and ensure clear visibility of coverage and risks.
• Write precise, actionable bug reports with reproducible steps and data; participate in triage.
• Contribute to code reviews and test-design discussions to elevate QA practices.

Skills

Python

Rust

Go

Test automation

QA engineering

Data pipelines

Metrics analysis

Debugging

Education

BS in Computer Science

MS/PhD in CS

Tools

Jenkins

Git

CI/CD

Test harnesses

Job description

Company Overview

Deepgram is the leading platform underpinning the emerging trillion-dollar Voice AI economy, providing real-time APIs for speech-to-text (STT), text-to-speech (TTS), and building production-grade voice agents at scale. More than 200,000 developers and 1,300+ organizations build voice offerings that are ‘Powered by Deepgram’, including Twilio, Cloudflare, Sierra, Decagon, Vapi, Daily, Cresta, Granola, and Jack in the Box. Deepgram’s voice-native foundation models are accessed through cloud APIs or as self-hosted and on-premises software, with unmatched accuracy, low latency, and cost efficiency. Backed by a recent Series C led by leading global investors and strategic partners, Deepgram has processed over 50,000 years of audio and transcribed more than 1 trillion words. There is no organization in the world that understands voice better than Deepgram.

Company Operating Rhythm

At Deepgram, we expect an AI-first mindset—AI use and comfort aren’t optional, they’re core to how we operate, innovate, and measure performance.

Every team member who works at Deepgram is expected to actively use and experiment with advanced AI tools, and even build your own into your everyday work. We measure how effectively AI is applied to deliver results, and consistent, creative use of the latest AI capabilities is key to success here. Candidates should be comfortable adopting new models and modes quickly, integrating AI into their workflows, and continuously pushing the boundaries of what these technologies can do.

Additionally, we move at the pace of AI. Change is rapid, and you can expect your day-to-day work to evolve just as quickly. This may not be the right role if you’re not excited to experiment, adapt, think on your feet, and learn constantly, or if you’re seeking something highly prescriptive with a traditional 9-to-5.

The Opportunity

Deepgram is looking for a Software Test Engineer to design, build, and maintain automated test frameworks and exploratory test suites across our products, models, APIs, and data platforms. You enjoy breaking systems, probing edge cases, testing real-world and adversarial inputs, and automating repeatable validation so regressions are caught quickly.

You translate product requirements and model metrics into automated regression tests, evaluation pipelines, data-quality gates, load tests, and release criteria. You partner with QA, Research, Product, Data, and Engineering to plan testing, execute human and automated evaluations, support user acceptance testing, and communicate risks clearly.

When you find an issue, you provide precise reproduction steps, inputs, parameters, expected and actual results, and supporting data. What gets you excited? Building scalable automation that gives Deepgram confidence that its products, models, and data workflows work reliably for customers.

What You'll Do

• Define and execute well-designed test plans across Deepgram's products, APIs, SDKs, model-powered features, and data platforms, ensuring production software is robust, reliable, and performs well.

• Design, build, and maintain automated test suites and frameworks for functional, integration, end-to-end, regression, API, browser, and service-level testing across batch and streaming workflows.

• Translate product requirements and customer acceptance criteria into clear test strategies, repeatable test cases, and enforceable release gates.

• Build and maintain representative, customer-focused, and adversarial test datasets, fixtures, and test environments that exercise real-world inputs, edge cases, failure modes, and system limits.

• Validate model-powered behavior—including speech-to-text, text-to-speech, and other AI features—using appropriate metrics, expected outputs, human review, and regression coverage, while partnering with Research and model-evaluation specialists as needed.

• Build testing infrastructure, including test harnesses, reusable scripts, test-data tooling, result-aggregation pipelines, dashboards, and visualizations that make quality signals easy to understand and act on.

• Integrate automated tests, quality checks, canaries, and release validation into CI/CD so regressions are detected continuously rather than through manual testing alone.

• Partner with Engineering, Product, Research, Data, Infrastructure, and DevOps to understand system behavior, dependencies, variations, performance limits, and deployment risks, and to establish appropriate test coverage.

• Test data ingestion, processing, annotation, and quality-control workflows, validating data integrity, completeness, representativeness, deduplication, leakage, and downstream readiness.

• Execute staging and production validation, load and reliability testing, cross-browser and customer-workflow testing, and user acceptance testing in partnership with internal stakeholders and customer QA teams.

• Maintain and improve the test-case repository, automation coverage, test documentation, and release-readiness reporting so teams have a clear view of what was tested, what passed, and what remains risky.

• Write precise, actionable bug reports with reproducible steps, inputs, parameters, expected and actual results, logs or artifacts, and clear severity; participate in triage and elevate issues when necessary.

• Help raise the bar through code reviews, test-design reviews, technical discussions, and strong engineering, automation, and QA practices.

What We're Looking For

• BS, MS, or PhD in Computer Science, AI, Applied Math, or a related field, or equivalent experience.

• 5+ years of professional software or QA engineering experience, with a track record of shipping test infrastructure or evaluation systems (senior candidates with significantly deeper experience welcome).

• Solid backend/scripting experience in a language such as Python, Rust, Go, or similar.

• Experience designing and building automated test pipelines, evaluation frameworks, or data-processing systems.

• Strong analytical skills and comfort reasoning about metrics, thresholds, and statistical variation in results — able to distinguish real regressions from noise.

• Ability to take charge of ambiguous technical challenges and communicate effectively across research, engineering, and product teams.

Nice to Have / Ways to Stand Out

• Hands-on experience testing or evaluating modern AI systems such as LLMs, RAG pipelines, agents, or multimodal models, including analyzing model behavior and failure modes.

• Experience with voice, audio, speech recognition, or real-time systems, and familiarity with metrics such as WER, MOS, latency, and time-to-first-byte.

• Experience building or improving test, evaluation, benchmarking, or ML infrastructure used by multiple teams or external users.

• A strong appreciation for test and evaluation quality, including correctness, reproducibility, determinism, and consistency across environments.

• Experience building test tooling for React Native, mobile applications, or other cross-platform environments that extends validation beyond the desktop.

• Familiarity with cloud infrastructure, containers, ephemer ...