AI Quality Assurance Engineer for Healthcare AI

Bask Health

Oregon

Hybrid

USD 120,000 - 170,000

Full time

7 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Bask Health seeks an AI Evaluation Engineer to verify AI-powered features across the platform, ensuring accuracy, reliability, and safety before and after production. You will design evaluation workflows, build datasets, and automate tests for language models, retrieval systems, and tool-calling.

Collaborating with AI, backend, product, and QA teams, you will translate product requirements into repeatable evaluations, document failures, and help raise quality standards.

Qualifications

  • 3+ years of professional software engineering or AI testing experience.
  • Experience developing automated tests for production software.
  • Strong programming skills in Python, TypeScript, JavaScript, or comparable languages.
  • Familiarity with large language models and AI-powered applications.
  • Understanding of software testing methodologies and test case design.
  • Experience building automated testing scripts, frameworks, or validation tools.
  • Familiarity with API testing and backend application integrations.
  • Understanding of prompt engineering and AI model failure modes.

Responsibilities

  • Design and implement evaluation workflows for AI-powered applications.
  • Build automated tests for large language model outputs and AI application behavior.
  • Develop evaluation datasets representing realistic user requests and product workflows.
  • Create test scenarios for patient-facing, merchant-facing, and internal AI features.
  • Implement checks for response accuracy, relevance, completeness, and consistency.
  • Evaluate AI-generated content for unsupported claims and misleading information.
  • Test structured model outputs for schema compliance and application compatibility.
  • Build evaluations for AI tool calling and authorized action execution.
  • Validate that AI assistants respect application permissions and tenant boundaries.

Skills

Python
TypeScript
JavaScript
API testing
CI/CD

Tools

LangSmith
OpenAI API
Playwright
Pytest

Job description

Bask Health seeks an AI Evaluation Engineer to verify AI-powered features across the platform, ensuring accuracy, reliability, and safety before and after production. You will design evaluation workflows, build datasets, and automate tests for language models, retrieval systems, and tool-calling.

Collaborating with AI, backend, product, and QA teams, you will translate product requirements into repeatable evaluations, document failures, and help raise quality standards.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Healthcare AI Quality Assurance Engineer
Healthcare AI Quality Assurance Engineer

Bask Health • United States

Remote
USD 120,000 - 180,000
AI Evaluation & Quality Assurance Engineer
AI Evaluation & Quality Assurance Engineer

Bask Health • New York (NY)

On-site
USD 120,000 - 170,000
AI Quality & Evaluation Engineer
AI Quality & Evaluation Engineer

Bask Health • New York (NY)

On-site
USD 120,000 - 190,000
AI Quality & Evaluation Engineer — Backend Tools
AI Quality & Evaluation Engineer — Backend Tools

Bask Health • United States

Remote
USD 120,000 - 180,000
AI Reliability & Evaluation Engineer (Healthcare)
AI Reliability & Evaluation Engineer (Healthcare)

Bask Health • Oregon

Hybrid
USD 120,000 - 180,000
AI Evaluation & Reliability Engineer
AI Evaluation & Reliability Engineer

Bask Health • United States

Remote
USD 140,000 - 200,000
AI Reliability Engineer — Evaluation & Testing
AI Reliability Engineer — Evaluation & Testing

Bask Health • New York (NY)

On-site
USD 120,000 - 190,000
AI Quality & Evaluation Product Leader, Healthcare
AI Quality & Evaluation Product Leader, Healthcare

wk • United States

Remote
USD 120,000 - 190,000
Senior ML Engineer, AI Evaluation & Reliability
Senior ML Engineer, AI Evaluation & Reliability

Bask Health • United States

Remote
USD 150,000 - 210,000
Senior ML Reliability Engineer - AI Evaluation & HealthTech
Senior ML Reliability Engineer - AI Evaluation & HealthTech

Bask Health • New York (NY)

On-site
USD 180,000 - 240,000