AI/ML Quality Engineer — LLM Evaluation & QA Automation

INSPYR Solutions

Cupertino (CA)

Hybrid

USD 91,000 - 105,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Remote-friendly schedule

Job summary

INSPYR Solutions is seeking an AI/ML Quality Engineer to join our team in Cupertino. You will design and execute manual and automated tests for GenAI platforms, validate LLM outputs, and ensure end-to-end system reliability across data pipelines and cloud workloads.

You will build Python-based test tools, apply ML evaluation strategies with LLM-as-a-Judge concepts, and collaborate with localization teams to ensure multilingual quality.

Qualifications

  • 3+ years of QA Automation engineering experience.
  • Experience with Playwright for automated testing including sign-in and SSO flows.
  • Design tests that validate output accuracy and system behavior across user flows.
  • Experience with Python API testing using requests, Pytest, and CI/CD in cloud environments.
  • Hands-on ML/LLM evaluation experience using LLM-as-a-Judge models.

Responsibilities

  • Design and execute manual and automated test cases for GenAI platforms and chatbot systems.
  • Build Python-based test automation frameworks for backend services and ML pipelines.
  • Develop ML evaluation strategies to assess response accuracy, relevance, and quality.
  • Collaborate with localization teams to ensure multilingual output quality.
  • Support data annotation workflows and feedback loops for model improvement.
  • Utilize frameworks to evaluate LLM outputs and identify optimization opportunities.
  • Conduct end-to-end testing from data ingestion to user-facing output.
  • Validate stability across GCP components and Kubernetes deployments.
  • Collaborate with DevOps/ML engineers to triage bugs and improve performance.

Skills

QA Automation
LLM Evaluation
Python API testing
Pytest
CI/CD
GCP
Localization testing
Multilingual evaluation

Tools

Playwright

Job description

INSPYR Solutions is seeking an AI/ML Quality Engineer to join our team in Cupertino. You will design and execute manual and automated tests for GenAI platforms, validate LLM outputs, and ensure end-to-end system reliability across data pipelines and cloud workloads.

You will build Python-based test tools, apply ML evaluation strategies with LLM-as-a-Judge concepts, and collaborate with localization teams to ensure multilingual quality.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI/ML Quality Engineer
AI/ML Quality Engineer

INSPYR Solutions • Cupertino (CA)

Hybrid
USD 91,000 - 105,000
Remote-friendly schedule
Senior LLM Inference QA & Automation Engineer
Senior LLM Inference QA & Automation Engineer

Apple, Inc. • Seattle (WA)

On-site
USD 140,000 - 190,000
AI QA Engineer for LLM & GenAI Testing & Automation
AI QA Engineer for LLM & GenAI Testing & Automation

Compunnel Inc. • New York (NY)

Hybrid
USD 140,000 - 190,000
Senior AI Quality Leader
Senior AI Quality Leader

LILT AI • City of Syracuse (NY)

Hybrid
USD 150,000 - 190,000
AI Quality Engineer: LLM Testing & Validation
AI Quality Engineer: LLM Testing & Validation

Global Business Ser. 4u Inc. • Plano (TX)

On-site
USD 120,000 - 180,000
Remote AI QA Engineer for LLMs & Autonomous Agents
Remote AI QA Engineer for LLMs & Autonomous Agents

web • United States

Remote
USD 120,000 - 180,000
AIML - Sr. LLM Inference Quality Engineer
AIML - Sr. LLM Inference Quality Engineer

Apple, Inc. • Seattle (WA)

On-site
USD 140,000 - 190,000
LLM & AI QA Engineer: Reliability & Safety Tester
LLM & AI QA Engineer: Reliability & Safety Tester

Synoptek • United States

Remote
USD 90,000 - 120,000
AI Test Engineer (LLM and AI QA)
AI Test Engineer (LLM and AI QA)

Synoptek • United States

Remote
USD 90,000 - 120,000
Remote AI Evaluation Engineer — LLM Quality & Production
Remote AI Evaluation Engineer — LLM Quality & Production

Sedgwick • Tallahassee (FL)

Remote
USD 120,000 - 180,000