Staff AI Engineer - QA

Harriscomputer

United States

Remote

USD 120,000 - 160,000

Full time

4 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

medical benefits
vacation & personal days
employee stock ownership
RRSP/401k matching
flexible work arrangements

Job summary

Harris Computer is seeking a Staff AI Engineer focused on Quality Assurance to lead the testing and reliability of healthcare AI/ML solutions. This remote role is open to candidates in the United States or Canada, offering a path to shape how AI quality is defined and measured across clinical workflows.

The position requires extensive experience in software/ML engineering with a proven ability to design end-to-end systems, build evaluation frameworks, and establish rigorous metrics for accuracy

Qualifications

  • 7+ years in software or ML engineering with 3+ years in evaluation or QA for AI systems.
  • Proven track record building end-to-end systems from prototype to production.
  • Experience running evaluation systems in production, not just one-off analyses.
  • 2+ years evaluating large language models (LLMs) in production environments.
  • Strong statistical skills: sampling, confidence intervals, error analysis.
  • 1+ year designing evaluation sets for document extraction, classification, or structured output.
  • Deep knowledge of XAI techniques and AI-powered tools.

Responsibilities

  • Build the core AI system end-to-end, from service layer to evaluation infrastructure.
  • Design and maintain evaluation datasets with clinician-reviewed answers.
  • Create error metrics weighted by severity for clinically/financially impactful mistakes.
  • Develop production monitoring for accuracy, overrides, and safety issues with alerts.
  • Write adversarial tests for AI systems, including hidden instructions in documents.
  • Collaborate with product and clinical leads to define accuracy thresholds for capability expansion.

Skills

System design
Quality testing
Statistical analysis
Explainable AI (XAI)

Tools

SHAP
LIME
Claude
Great Expectations

Job description

Harris Computer is hiring a Staff AI Engineer to join their, expanding team! We’re seeking a seasoned expert to pioneer the quality and reliability of our cutting‑edge AI and machine learning solutions for healthcare. As a Staff AI Engineer focused on Quality Assurance, you will be the technical authority responsible for ensuring our AI systems are robust, fair, accurate, and safe. This is a unique opportunity to define the future of AI quality in a domain where it matters most, mentoring a talented team and establishing the gold standard for testing intelligent systems that impact patient care. This is a full‑time, remote position open to candidates in the U.S. or Canada.

What your impact will be:
  • Build the core AI system end‑to‑end, from the service layer and data pipelines to the evaluation infrastructure, and own its architecture.
  • Design and build the evaluation infrastructure, including dataset management, automated test harnesses, and replay of AI systems against historical workflows.
  • Build and maintain evaluation datasets with clinician‑reviewed correct answers, ensuring they evolve as our systems change.
  • Create error metrics weighted by severity, so mistakes with clinical or financial impact are prioritized accordingly.
  • Build production monitoring for accuracy, overrides, and safety issues, with actionable alerts for the engineering team.
  • Write adversarial tests for AI systems, including hidden or malicious instructions in documents and invented content.
  • Work with product and clinical leads to define accuracy thresholds and the criteria for expanding or pulling back an AI capability.
What we’re looking for:
  • 7 or more years in software or machine learning engineering, with at least 3 years focused on evaluation, testing, or quality for AI systems.
  • A track record of designing and building complete systems from scratch, taking them from prototype to production.
  • A track record of building and running evaluation systems that teams use in production, not just for one‑off analyses.
  • At least 2 years of experience evaluating large language model (LLM) systems in a production environment.
  • Strong statistical skills for measuring accuracy, including sampling, confidence intervals, and error analysis.
  • 1 year + of experience designing evaluation sets for document extraction, classification, or structured output.
  • Deep knowledge of explainable AI (XAI) techniques and AI‑powered tools (e.g., SHAP, LIME, Claude).
  • Clear written communication, including design documents, and the ability to explain complex trade‑offs to clinicians, engineers, and executives.
What would make you stand out:
  • Experience working within the healthcare or another highly regulated industry.
  • Familiarity with AI‑specific testing techniques like metamorphic testing, adversarial testing, or data validation with tools like Great Expectations.
  • A history of mentoring engineers and influencing technical direction across multiple teams.
What we offer:
  • Plenty of opportunities to grow your career Comprehensive medical, dental, and vision benefits 3 weeks of vacation plus 5 personal days to recharge Employee stock ownership, RRSP program, 401k matching A chance to give back through community involvement Flexible work arrangements to suit your lifestyle Salary range: 120,000-160,000 depending on experience and qualifications.
About Harris Computer:

Harris provides mission critical software solutions for the Public Sector, Healthcare, Utilities and Private Sector verticals throughout North America, Europe, Asia and Australia. Working for Harris is the perfect opportunity to fulfill your professional goals as well as achieve your personal dreams! Our employees enjoy a casual work environment that offers comfort while providing superior service to our customers. We offer a comprehensive benefit package as well as other additional “Perks”! We empower our employees to make a difference We have an award‑winning culture We offer opportunity to learn We are financially strong and we are owned by the largest software company in Canada (CSI) We have fun! #LI-remote

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff AI Engineer - QA
Staff AI Engineer - QA

Harris Computer Corp. • United States

Remote
USD 120,000 - 160,000
Medical, dental, vision coverage
3 weeks vacation + 5 personal days
Employee stock ownership
+3
Staff AI Engineer - QA
Staff AI Engineer - QA

N. Harris Computer Corporation - USA • Raleigh (NC)

Remote
USD 120,000 - 160,000
Medical, dental, and vision benefits
Flexible work arrangements
Career growth opportunities
+2
Staff AI Engineer - QA
Staff AI Engineer - QA

Harris Computer Corp. • Texas

Remote
USD 120,000 - 160,000
Career growth
Medical/Dental/Vision
Vacation and personal days
+4
Staff AI Engineer - QA
Staff AI Engineer - QA

Harris Computer Corp. • New York (NY)

Remote
USD 120,000 - 160,000
Growth opportunities
Medical benefits
Dental benefits
+7
Staff AI Engineer - QA
Staff AI Engineer - QA

Harris Computer Corp. • Virginia

Remote
USD 120,000 - 160,000
Career growth
Medical, dental, vision benefits
Stock ownership and retirement plans
+2
Staff AI Engineer - QA
Staff AI Engineer - QA

Harris Computer Corp. • South Carolina

Remote
USD 120,000 - 160,000
Career growth opportunities
Medical, dental, and vision benefits
3 weeks vacation + 5 personal days
+1
Staff AI Engineer - QA
Staff AI Engineer - QA

Harris Computer Corp. • Michigan

Remote
USD 120,000 - 160,000
Career growth
Comprehensive benefits
Vacation + personal days
+1
Staff AI Engineer - QA
Staff AI Engineer - QA

Harris Computer Corp. • Tennessee

Remote
USD 120,000 - 160,000
Career growth
Medical, dental, vision benefits
Vacation time (3 weeks) and personal 5
+2
Staff AI Engineer - QA
Staff AI Engineer - QA

Harris Computer Corp. • New Mexico

Remote
USD 120,000 - 160,000
Medical benefits
Dental and vision coverage
Vacation plus personal days
+3
Staff AI Engineer - DevOps
Staff AI Engineer - DevOps

Harriscomputer • United States

Remote
USD 120,000 - 160,000
Plenty of opportunities to grow your职业
Comprehensive medical, dental, and视力保险
3 weeks of vacation plus 5 personal
+4