AI QA Engineer: LLM Testing & Automation

Intone Inc; Bellevue (WA); Hybrid; USD 120,000 - 160,000; Full time

Job summary

Intone Inc. in Bellevue, WA is seeking a QA Engineer specialized in AI products for a 6-month contract. The role is hybrid and requires local candidates for in-person collaboration.

You will design and execute tests for AI/ML features, build automated suites, and evaluate model outputs for accuracy, bias, and safety, while collaborating with data scientists and ML engineers to ensure robust quality across pipelines and deployments.

Qualifications

• 5+ years of QA and testing experience.
• Hands-on experience testing LLM and generative AI products.
• Practical experience with AI/LLM evaluation frameworks such as Ragas, DeepEval, LangSmith, Promptfoo, OpenAI Evals or TruLens.

Responsibilities

• Design and execute test plans for AI/ML-driven features, including model outputs and prompts.
• Build and maintain automated test suites covering functional, regression, integration and API testing.
• Evaluate model outputs for accuracy, bias, hallucination and edge-case failures.
• Develop evaluation frameworks and golden datasets to benchmark model performance over time.
• Test prompt changes, model version upgrades, and fine-tuning outputs for regressions.
• Perform adversarial and red-team style testing to surface safety and robustness issues.
• Validate data pipelines feeding into AI models, including data quality and drift detection.
• Collaborate with data scientists and ML engineers to define acceptance criteria and quality metrics.
• Test latency, scalability and reliability of AI services under load.
• Contribute to CI/CD pipelines with automated and model-evaluation tests.
• Design test cases for LLM-based products such as chatbots and RAG systems.

Skills

QA testing

LLM testing

Python

SQL

Test automation

Tools

Ragas

DeepEval

LangSmith

PromptFoo

OpenAI Evals

TruLens

Job description

Intone Inc. in Bellevue, WA is seeking a QA Engineer specialized in AI products for a 6-month contract. The role is hybrid and requires local candidates for in-person collaboration.

You will design and execute tests for AI/ML features, build automated suites, and evaluate model outputs for accuracy, bias, and safety, while collaborating with data scientists and ML engineers to ensure robust quality across pipelines and deployments.