Staff AI Quality Engineer – Remote Healthcare

N. Harris Computer Corporation - USA

Raleigh (NC)

Remote

USD 120,000 - 160,000

Full time

4 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Medical, dental, and vision benefits
Flexible work arrangements
Career growth opportunities
Employee stock ownership
401k matching

Job summary

Harris Computer is seeking a Staff AI Engineer focused on Quality Assurance to drive the reliability and fairness of AI/ML solutions for healthcare. You will mentor a team, define testing gold standards, and own the evaluation framework from data pipelines to deployment.

We value a seasoned engineer with strong statistical skills and a proven track record in evaluating LLMs in production, adapting cutting‑edge tools to improve safety and accuracy for patient care. Remote within the US/Canada.

Qualifications

  • 7+ years in software or machine learning engineering, with 3+ years in evaluation or quality for AI systems.
  • Proven track record building end‑to‑end systems from prototype to production.
  • Experience building/reviewing production evaluation systems used by teams.
  • At least 2 years evaluating large language models (LLMs) in production.
  • Strong statistical skills for accuracy measurement, sampling, and error analysis.
  • Experience designing evaluation sets for document extraction, classification, or structured output.
  • Ability to communicate complex trade‑offs clearly to clinicians, engineers, and executives.
  • Familiarity with AI testing techniques and tools (e.g., metamorphic testing, data validation).

Responsibilities

  • Build the core AI system end‑to‑end, from service layer to evaluation infrastructure and own architecture.
  • Design and build evaluation infrastructure, including dataset management and automated test harnesses.
  • Maintain evaluation datasets with clinician‑reviewed answers and evolve them with system changes.
  • Create error metrics weighted by severity to prioritize clinically or financially impactful issues.
  • Develop production monitoring for accuracy, overrides, and safety with actionable alerts.
  • Write adversarial tests for AI systems, including hidden or malicious content scenarios.
  • Collaborate with product and clinical leads to set accuracy thresholds and expansion criteria.
  • Curate and leverage emerging AI tools to improve workflows and efficiency.

Skills

Software engineering
Machine learning
Evaluation and testing
Statistical analysis
LLM evaluation

Tools

SHAP
LIME
Claude

Job description

Harris Computer is seeking a Staff AI Engineer focused on Quality Assurance to drive the reliability and fairness of AI/ML solutions for healthcare. You will mentor a team, define testing gold standards, and own the evaluation framework from data pipelines to deployment.

We value a seasoned engineer with strong statistical skills and a proven track record in evaluating LLMs in production, adapting cutting‑edge tools to improve safety and accuracy for patient care. Remote within the US/Canada.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Quality Engineer – Healthcare (Remote)
Senior AI Quality Engineer – Healthcare (Remote)

Harris Computer Corp. • Virginia

Remote
USD 120,000 - 160,000
Career growth
Medical, dental, vision benefits
Stock ownership and retirement plans
+2
Lead AI Quality Assurance Engineer – Remote Healthcare
Lead AI Quality Assurance Engineer – Remote Healthcare

Harris Computer Corp. • New York (NY)

Remote
USD 120,000 - 160,000
Growth opportunities
Medical benefits
Dental benefits
+7
Staff AI QA Engineer – Healthcare, Remote
Staff AI QA Engineer – Healthcare, Remote

Harris Computer Corp. • United States

Remote
USD 120,000 - 160,000
Medical, dental, vision coverage
3 weeks vacation + 5 personal days
Employee stock ownership
+3
Staff AI QA Engineer - Remote, Equity & Growth
Staff AI QA Engineer - Remote, Equity & Growth

Harriscomputer • United States

Remote
USD 120,000 - 160,000
medical benefits
vacation & personal days
employee stock ownership
+2
Staff AI QA Engineer – Healthcare Quality Lead (Remote)
Staff AI QA Engineer – Healthcare Quality Lead (Remote)

Harris Computer Corp. • Texas

Remote
USD 120,000 - 160,000
Career growth
Medical/Dental/Vision
Vacation and personal days
+4
Remote Staff AI QA Engineer - Healthcare AI Quality Leader
Remote Staff AI QA Engineer - Healthcare AI Quality Leader

Harris Computer Corp. • Michigan

Remote
USD 120,000 - 160,000
Career growth
Comprehensive benefits
Vacation + personal days
+1
Staff AI QA Engineer - Healthcare, Remote, High Impact
Staff AI QA Engineer - Healthcare, Remote, High Impact

Harris Computer Corp. • New Mexico

Remote
USD 120,000 - 160,000
Medical benefits
Dental and vision coverage
Vacation plus personal days
+3
Staff AI QA Engineer – Healthcare AI Quality & Reliability (Remote)
Staff AI QA Engineer – Healthcare AI Quality & Reliability (Remote)

Harris Computer Corp. • South Carolina

Remote
USD 120,000 - 160,000
Career growth opportunities
Medical, dental, and vision benefits
3 weeks vacation + 5 personal days
+1
Remote Senior AI QA Engineer — Healthcare
Remote Senior AI QA Engineer — Healthcare

Harris Computer Corp. • Tennessee

Remote
USD 120,000 - 160,000
Career growth
Medical, dental, vision benefits
Vacation time (3 weeks) and personal 5
+2
Staff AI Engineer - QA
Staff AI Engineer - QA

Harris Computer Corp. • United States

Remote
USD 120,000 - 160,000
Medical, dental, vision coverage
3 weeks vacation + 5 personal days
Employee stock ownership
+3