AI Evaluation & Quality Assurance Engineer

Bask Health

New York (NY)

On-site

USD 120,000 - 170,000

Full time

6 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Bask Health is seeking an AI Evaluation Engineer to ensure AI-powered features are accurate, reliable, and compliant before production and during operations. You will design evaluation workflows, build automated tests for LLM outputs, and develop datasets representing realistic user scenarios across patient-facing, merchant-facing, and internal AI features.

You will collaborate with applied AI engineers, ML engineers, backend teams, and product managers to define expected behavior, implement

Qualifications

  • 3+ years of professional experience in software engineering, AI engineering, test automation, or a related field.
  • Experience developing automated tests for production software applications.
  • Strong programming skills in Python, TypeScript, JavaScript, or comparable languages.
  • Familiarity with large language models and modern AI-powered applications.
  • Understanding of software testing methodologies and test case design.
  • Experience building automated testing scripts, frameworks, or validation tools.
  • Familiarity with API testing and backend application integrations.
  • Understanding of prompt engineering and common AI model failure modes.
  • Ability to evaluate AI-generated responses against expected requirements.
  • Experience working with structured data and validating application outputs.
  • Familiarity with regression testing and software release processes.
  • Experience debugging application failures and reproducing technical issues.
  • Understanding of authentication, authorization, and secure application behavior.
  • Familiarity with databases and application data models.
  • Experience working with Git and collaborative software development workflows.
  • Understanding of continuous integration and automated testing pipelines.
  • Familiarity with logging, monitoring, and production troubleshooting.
  • Ability to create clear, reproducible test cases.
  • Strong analytical thinking and attention to detail.

Responsibilities

  • Design and implement evaluation workflows for AI-powered applications across Bask Health.
  • Build automated tests for large language model outputs and AI application behavior.
  • Develop evaluation datasets that represent realistic user requests and product workflows.
  • Create test scenarios for patient-facing, merchant-facing, and internal AI features.
  • Implement checks for response accuracy, relevance, completeness, and consistency.
  • Evaluate AI-generated content for unsupported claims and misleading information.
  • Test structured model outputs for schema compliance and application compatibility.
  • Build evaluations for AI tool calling and authorized action execution.
  • Validate that AI assistants respect application permissions and tenant boundaries.
  • Test retrieval-augmented generation systems for grounding and information relevance.
  • Develop regression tests that identify unexpected changes in AI behavior.
  • Integrate evaluation checks into engineering development and release workflows.
  • Create automated comparison tests for prompts, model versions, and configuration changes.
  • Develop scripts and utilities that simplify AI evaluation execution.
  • Build reporting tools that summarize evaluation results and failure patterns.
  • Investigate failed evaluations and help engineers reproduce underlying issues.
  • Work with product managers to define expected AI behavior and acceptance criteria.
  • Collaborate with applied AI engineers to improve prompts and application logic.
  • Test AI applications against unusual inputs, ambiguous requests, and difficult edge cases.
  • Develop evaluation cases for prompt injection and unauthorized tool-use attempts.

Skills

AI evaluation
Python
TypeScript
JavaScript
API testing
CI/CD pipelines
Git workflows
Regression testing

Tools

Cursor
Claude Code
Playwright
Jest
Vitest
Pytest

Job description

Bask Health is seeking an AI Evaluation Engineer to ensure AI-powered features are accurate, reliable, and compliant before production and during operations. You will design evaluation workflows, build automated tests for LLM outputs, and develop datasets representing realistic user scenarios across patient-facing, merchant-facing, and internal AI features.

You will collaborate with applied AI engineers, ML engineers, backend teams, and product managers to define expected behavior, implement

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Healthcare AI Quality Assurance Engineer
Healthcare AI Quality Assurance Engineer

Bask Health • United States

Remote
USD 120,000 - 180,000
AI Quality & Evaluation Engineer
AI Quality & Evaluation Engineer

Bask Health • New York (NY)

On-site
USD 120,000 - 190,000
AI Evaluation & Reliability Engineer
AI Evaluation & Reliability Engineer

Bask Health • United States

Remote
USD 140,000 - 200,000
AI Quality & Evaluation Engineer — Backend Tools
AI Quality & Evaluation Engineer — Backend Tools

Bask Health • United States

Remote
USD 120,000 - 180,000
Senior ML Engineer, AI Evaluation & Reliability
Senior ML Engineer, AI Evaluation & Reliability

Bask Health • United States

Remote
USD 150,000 - 210,000
AI Reliability Engineer — Evaluation & Testing
AI Reliability Engineer — Evaluation & Testing

Bask Health • New York (NY)

On-site
USD 120,000 - 190,000
Senior ML Reliability Engineer - AI Evaluation & HealthTech
Senior ML Reliability Engineer - AI Evaluation & HealthTech

Bask Health • New York (NY)

On-site
USD 180,000 - 240,000
AI Evaluation Platform Engineer
AI Evaluation Platform Engineer

Bask Health • New York (NY)

On-site
USD 140,000 - 190,000
Remote Senior LLM Evaluation & Reliability Engineer
Remote Senior LLM Evaluation & Reliability Engineer

Bask Health • United States

Remote
USD 160,000 - 220,000
AI Evaluation Architect
AI Evaluation Architect

Nouvells • United States

Remote
USD 90,000 - 130,000