Healthcare AI Quality Assurance Engineer

Bask Health

United States

Remoto

USD 120.000 - 180.000

Tempo integral

Há 6 dias
Seja uma das primeiras pessoas a se candidatar

Uma candidatura completa em um minuto

Currículo e carta de apresentação personalizados, prontos para serem enviados.

Passe pelos filtros ATS

Resumo da vaga

Bask Health is seeking an AI Evaluation Engineer to validate AI-powered features across the platform, ensuring accuracy, reliability, and compliance before and after production releases.

You will design evaluation workflows, build automated tests, and develop datasets that reflect real user scenarios across patient-facing, merchant-facing, and internal AI features. This role emphasizes practical testing and collaboration with product and engineering teams.

Qualificações

  • 3+ years of professional experience in software engineering, AI engineering, test automation, quality engineering, or a closely related technical field.
  • Experience developing automated tests for production software applications.
  • Strong programming skills in Python, TypeScript, JavaScript, or comparable languages.
  • Familiarity with large language models and modern AI-powered applications.
  • Understanding of software testing methodologies and test case design.
  • Experience building automated testing scripts, frameworks, or validation tools.
  • Familiarity with API testing and backend application integrations.
  • Understanding of prompt engineering and common AI model failure modes.
  • Ability to evaluate AI-generated responses against expected requirements.
  • Experience working with structured data and validating application outputs.
  • Familiarity with regression testing and software release processes.
  • Experience debugging application failures and reproducing technical issues.
  • Understanding of authentication, authorization, and secure application behavior.
  • Familiarity with databases and application data models.
  • Experience with Git and collaborative software development workflows.
  • Understanding of continuous integration and automated testing pipelines.
  • Familiarity with logging, monitoring, and production troubleshooting.
  • Ability to create clear, reproducible test cases.
  • Strong analytical thinking and attention to detail.
  • Ability to distinguish application defects from model behavior limitations.
  • Experience documenting technical issues and communicating findings.
  • Familiarity with cloud-based application environments.
  • Ability to collaborate effectively with engineers, product managers, and other stakeholders.
  • Strong problem-solving skills and a quality-focused engineering mindset.
  • Ability to independently own evaluation and test automation projects.
  • Nice to Have

Responsabilidades

  • Design and implement evaluation workflows for AI-powered applications across Bask Health.
  • Build automated tests for large language model outputs and AI application behavior.
  • Develop evaluation datasets that represent realistic user requests and product workflows.
  • Create test scenarios for patient-facing, merchant-facing, and internal AI features.
  • Implement checks for response accuracy, relevance, completeness, and consistency.
  • Evaluate AI-generated content for unsupported claims and misleading information.
  • Test structured model outputs for schema compliance and application compatibility.
  • Build evaluations for AI tool calling and authorized action execution.
  • Validate that AI assistants respect application permissions and tenant boundaries.
  • Test retrieval-augmented generation systems for grounding and information relevance.
  • Develop regression tests that identify unexpected changes in AI behavior.
  • Integrate evaluation checks into engineering development and release workflows.
  • Create automated comparison tests for prompts, model versions, and configuration changes.
  • Develop scripts and utilities that simplify AI evaluation execution.
  • Build reporting tools that summarize evaluation results and failure patterns.
  • Investigate failed evaluations and help engineers reproduce underlying issues.
  • Work with product managers to define expected AI behavior and acceptance criteria.
  • Collaborate with applied AI engineers to improve prompts and application logic.
  • Test AI applications against unusual inputs, ambiguous requests, and difficult edge cases.
  • Develop evaluation cases for prompt injection and unauthorized tool-use attempts.
  • Support human review workflows for outputs that require subjective assessment.
  • Track recurring AI failures and help prioritize reliability improvements.
  • Monitor production evaluation signals and investigate quality regressions.
  • Document test coverage, evaluation procedures, and known limitations.
  • Improve evaluation efficiency, repeatability, and maintainability.
  • Participate in code reviews, engineering planning, and technical discussions.
  • Support controlled testing and rollout of new AI-powered features.
  • Own assigned AI evaluation projects from test design through implementation, reporting, and ongoing improvement.

Conhecimentos

AI testing
LLM evaluation
Test automation
Python/TypeScript
Regression testing
API testing
Data validation

Ferramentas

Playwright
Jest / Vitest
Pytest
Git
CI/CD pipelines

Descrição da vaga

Bask Health is seeking an AI Evaluation Engineer to validate AI-powered features across the platform, ensuring accuracy, reliability, and compliance before and after production releases.

You will design evaluation workflows, build automated tests, and develop datasets that reflect real user scenarios across patient-facing, merchant-facing, and internal AI features. This role emphasizes practical testing and collaboration with product and engineering teams.

Obtenha sua revisão de currículo grátis e confidencial.

ou arraste e solte seu arquivo aqui.

Similar jobs

Vagas similares que valem a pena comparar

AI Evaluation & Quality Assurance Engineer
AI Evaluation & Quality Assurance Engineer

Bask Health • New York (NY)

Presencial
USD 120.000 - 170.000
AI Quality & Evaluation Engineer
AI Quality & Evaluation Engineer

Bask Health • New York (NY)

Presencial
USD 120.000 - 190.000
AI Evaluation & Reliability Engineer
AI Evaluation & Reliability Engineer

Bask Health • EUA

Remoto
USD 140.000 - 200.000
AI Quality & Evaluation Engineer — Backend Tools
AI Quality & Evaluation Engineer — Backend Tools

Bask Health • EUA

Remoto
USD 120.000 - 180.000
Senior ML Engineer, AI Evaluation & Reliability
Senior ML Engineer, AI Evaluation & Reliability

Bask Health • EUA

Remoto
USD 150.000 - 210.000
Senior ML Reliability Engineer - AI Evaluation & HealthTech
Senior ML Reliability Engineer - AI Evaluation & HealthTech

Bask Health • New York (NY)

Presencial
USD 180.000 - 240.000
Applied AI Engineer - Patient Experience & Navigation
Applied AI Engineer - Patient Experience & Navigation

Bask Health • EUA

Remoto
USD 120.000 - 180.000
AI Reliability Engineer — Evaluation & Testing
AI Reliability Engineer — Evaluation & Testing

Bask Health • New York (NY)

Presencial
USD 120.000 - 190.000
AI Evaluation Platform Engineer
AI Evaluation Platform Engineer

Bask Health • New York (NY)

Presencial
USD 140.000 - 190.000
Applied AI Engineer for Patient Experience & Onboarding
Applied AI Engineer for Patient Experience & Onboarding

Bask Health • New York (NY)

Presencial
USD 120.000 - 180.000