Staff AI QA Engineer – Healthcare, Remote

Harris Computer Corp.

United States

Remote

USD 120,000 - 160,000

Full time

5 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Medical, dental, vision coverage
3 weeks vacation + 5 personal days
Employee stock ownership
RRSP program
401k matching
Flexible work arrangements

Job summary

Harris Computer Corp. is seeking a Staff AI Engineer focused on Quality Assurance to lead the reliability and safety of our AI/ML healthcare solutions. You will architect and own the evaluation framework, data pipelines, and monitoring for production AI systems, coordinating with product and clinical leads to set accuracy thresholds.

The role emphasizes building robust evaluation datasets, incorporating clinician-reviewed answers, and creating targeted adversarial tests. Remote in the U.S.

Qualifications

  • 7+ years in software or ML engineering with at least 3 years focused on evaluation, testing, or quality for AI systems.
  • Proven track record of designing and building complete systems from scratch, taking them from prototype to production.
  • Experience building and running evaluation systems that teams use in production.
  • At least 2 years of experience evaluating large language model (LLM) systems in a production environment.
  • Strong statistical skills for measuring accuracy, including sampling, confidence intervals, and error analysis.
  • 1 year+ of experience designing evaluation sets for document extraction, classification, or structured output.
  • Deep knowledge of explainable AI (XAI) techniques and AI-powered tools (e.g., SHAP, LIME, Claude).
  • Clear written communication, including design documents, and the ability to explain complex trade-offs to clinicians, engineers, and executives.

Responsibilities

  • Build the core AI system end-to-end, from the service layer and data pipelines to the evaluation infrastructure, and own its architecture.
  • Design and build the evaluation infrastructure, including dataset management, automated test harnesses, and replay of AI systems against historical workflows.
  • Build and maintain evaluation datasets with clinician-reviewed correct answers, ensuring they evolve as our systems change.
  • Create error metrics weighted by severity, so mistakes with clinical or financial impact are prioritized accordingly.
  • Build production monitoring for accuracy, overrides, and safety issues, with actionable alerts for the engineering team.
  • Write adversarial tests for AI systems, including hidden or malicious instructions in documents and invented content.
  • Work with product and clinical leads to define accuracy thresholds and the criteria for expanding or pulling back an AI capability.

Skills

Senior software/ML engineering
AI evaluation/QA
Production eval systems
LLM evaluation (production)
Statistics (sampling, CIs, error)
Evaluation data design
XAI knowledge (SHAP, LIME, Claude)
Technical writing

Job description

Harris Computer Corp. is seeking a Staff AI Engineer focused on Quality Assurance to lead the reliability and safety of our AI/ML healthcare solutions. You will architect and own the evaluation framework, data pipelines, and monitoring for production AI systems, coordinating with product and clinical leads to set accuracy thresholds.

The role emphasizes building robust evaluation datasets, incorporating clinician-reviewed answers, and creating targeted adversarial tests. Remote in the U.S.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff AI QA Engineer – Healthcare Quality Lead (Remote)
Staff AI QA Engineer – Healthcare Quality Lead (Remote)

Harris Computer Corp. • Texas

Remote
USD 120,000 - 160,000
Career growth
Medical/Dental/Vision
Vacation and personal days
+4
Staff AI QA Engineer - Healthcare, Remote, High Impact
Staff AI QA Engineer - Healthcare, Remote, High Impact

Harris Computer Corp. • New Mexico

Remote
USD 120,000 - 160,000
Medical benefits
Dental and vision coverage
Vacation plus personal days
+3
Senior AI Quality Engineer – Healthcare (Remote)
Senior AI Quality Engineer – Healthcare (Remote)

Harris Computer Corp. • Virginia

Remote
USD 120,000 - 160,000
Career growth
Medical, dental, vision benefits
Stock ownership and retirement plans
+2
Staff AI Quality Engineer – Remote Healthcare
Staff AI Quality Engineer – Remote Healthcare

N. Harris Computer Corporation - USA • Raleigh (NC)

Remote
USD 120,000 - 160,000
Medical, dental, and vision benefits
Flexible work arrangements
Career growth opportunities
+2
Remote Senior AI QA Engineer — Healthcare
Remote Senior AI QA Engineer — Healthcare

Harris Computer Corp. • Tennessee

Remote
USD 120,000 - 160,000
Career growth
Medical, dental, vision benefits
Vacation time (3 weeks) and personal 5
+2
Staff AI QA Engineer – Healthcare AI Quality & Reliability (Remote)
Staff AI QA Engineer – Healthcare AI Quality & Reliability (Remote)

Harris Computer Corp. • South Carolina

Remote
USD 120,000 - 160,000
Career growth opportunities
Medical, dental, and vision benefits
3 weeks vacation + 5 personal days
+1
Remote Staff AI QA Engineer - Healthcare AI Quality Leader
Remote Staff AI QA Engineer - Healthcare AI Quality Leader

Harris Computer Corp. • Michigan

Remote
USD 120,000 - 160,000
Career growth
Comprehensive benefits
Vacation + personal days
+1
Lead AI Quality Assurance Engineer – Remote Healthcare
Lead AI Quality Assurance Engineer – Remote Healthcare

Harris Computer Corp. • New York (NY)

Remote
USD 120,000 - 160,000
Growth opportunities
Medical benefits
Dental benefits
+7
Staff AI QA Engineer - Remote, Equity & Growth
Staff AI QA Engineer - Remote, Equity & Growth

Harriscomputer • United States

Remote
USD 120,000 - 160,000
medical benefits
vacation & personal days
employee stock ownership
+2
Staff AI Engineer - QA
Staff AI Engineer - QA

Harriscomputer • United States

Remote
USD 120,000 - 160,000
medical benefits
vacation & personal days
employee stock ownership
+2