AI/ML Quality Engineer

INSPYR Solutions

Cupertino (CA)

Hybrid

USD 91.000 - 105.000

Vollzeit

14 Tage+

Erhalte eine Antwort von diesem Arbeitgeber

Dein Lebenslauf und dein Anschreiben, genau auf das zugeschnitten, was gesucht wird.

Schaffe es an den ATS-Filtern vorbei

Benefits dieser Stelle

Remote-friendly schedule

Zusammenfassung

INSPYR Solutions is seeking an AI/ML Quality Engineer to join our team in Cupertino. You will design and execute manual and automated tests for GenAI platforms, validate LLM outputs, and ensure end-to-end system reliability across data pipelines and cloud workloads.

You will build Python-based test tools, apply ML evaluation strategies with LLM-as-a-Judge concepts, and collaborate with localization teams to ensure multilingual quality.

Qualifikationen

  • 3+ years of QA Automation engineering experience.
  • Experience with Playwright for automated testing including sign-in and SSO flows.
  • Design tests that validate output accuracy and system behavior across user flows.
  • Experience with Python API testing using requests, Pytest, and CI/CD in cloud environments.
  • Hands-on ML/LLM evaluation experience using LLM-as-a-Judge models.

Aufgaben

  • Design and execute manual and automated test cases for GenAI platforms and chatbot systems.
  • Build Python-based test automation frameworks for backend services and ML pipelines.
  • Develop ML evaluation strategies to assess response accuracy, relevance, and quality.
  • Collaborate with localization teams to ensure multilingual output quality.
  • Support data annotation workflows and feedback loops for model improvement.
  • Utilize frameworks to evaluate LLM outputs and identify optimization opportunities.
  • Conduct end-to-end testing from data ingestion to user-facing output.
  • Validate stability across GCP components and Kubernetes deployments.
  • Collaborate with DevOps/ML engineers to triage bugs and improve performance.

Kenntnisse

QA Automation
LLM Evaluation
Python API testing
Pytest
CI/CD
GCP
Localization testing
Multilingual evaluation

Tools

Playwright

Jobbeschreibung

Title: AI/ML Quality Engineer
Location: Cupertino, CA (hybrid: Tues-Thurs in office)
Duration: 12+ month contract
Compensation: $66 - 76.00/hr. W2
Work Requirements: U.S. Citizen, GC Holders, or Authorized to Work in the U.S.
ABOUT THIS FEATURED OPPORTUNITY

The AI/ML Quality Engineer will join the Channel Sales and Operations team to help ensure the reliability and quality of the AI/ML-powered B2B chatbot and the foundational platforms supporting it. This role goes beyond traditional testing, with a focus on evaluating LLM outputs, improving model performance through structured feedback loops, and validating end-to-end system functionality, including cloud infrastructure and data pipelines.

THE OPPORTUNITY FOR YOU
  • Design and execute manual and automated test cases for GenAI platforms and chatbot systems.
  • Build and maintain Python-based test automation frameworks for backend services and ML pipeline validation.
  • Develop and execute ML evaluation strategies, leveraging LLM-as-a-Judge methodologies and models such as Gemini and Claude to assess response accuracy, relevance, factuality, and overall quality.
  • Collaborate with localization teams to conduct human evaluations, ensuring model outputs meet quality standards across different languages and markets.
  • Understand and support data annotation workflows, incorporating human feedback and evaluation results into continuous model improvement and fine-tuning.
  • Utilize RAGAS or similar frameworks to evaluate LLM outputs and identify opportunities for optimization.
  • Conduct end-to-end testing from data ingestion to user-facing output.
  • Validate system stability across GCP cloud components, including compute, storage, networking, and containers.
  • Collaborate with DevOps and ML engineers to triage bugs, improve model performance, and ensure system reliability.
  • Ensure test coverage spans multiple deployment environments, including Kubernetes clusters and cloud VMs.
KEY SUCCESS FACTORS
  • 3+ years of QA Automation Engineering experience
    • Experience with Playwright for automated application testing, including sign-in and SSO authentication flows
    • Ability to design tests that validate output accuracy and system behavior across different user flows
    • Experience with Python API testing using requests, Pytest, and integration into CI/CD and cloud environments.
  • Experience with ML/LLM Evaluation
    • Hands-on experience implementing LLM-as-a-Judge evaluation methodologies using models such as Gemini, Claude, or similar LLMs
    • Experience designing evaluation criteria and assessing LLM outputs for accuracy, relevance, factuality, and quality
    • Understanding of human evaluation and data annotation processes, including leveraging feedback loops to improve and fine-tune model performance
  • Experience with Cloud Platforms (GCP preferred)
    • Understanding of modern cloud architecture and how distributed components interconnect in production environments
NICE TO HAVES
  • Experience with multilingual LLM evaluation, localization testing, and human-in-the-loop evaluation workflows
About INSPYR Solutions

Technology is our focus and quality is our commitment. As a national expert in delivering flexible technology and talent solutions, we strategically align industry and technical expertise with our clients' business objectives and cultural needs. Our solutions are tailored to each client and include a wide variety of professional services, project, and talent solutions. By always striving for excellence and focusing on the human aspect of our business, we work seamlessly with our talent and clients to match the right solutions to the right opportunities. Learn more about us at inspyrsolutions.com.

INSPYR Solutions provides Equal Employment Opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability, genetics, or any other protected status. INSPYR Solutions complies with all applicable laws governing nondiscrimination in employment in every location in which the company has facilities. Applicants requiring reasonable accommodation during the application or interview process should contact HR@inspyrsolutions.com for assistance. Information collected and processed through your application with INSPYR Solutions (including any job applications you choose to submit) is subject to INSPYR Solutions' Privacy Policy and INSPYR Solutions' AI and Automated Employment Decision Tool Policy: https://www.inspyrsolutions.com/policies/. By submitting an application, you are consenting to being contacted by INSPYR Solutions through phone, email, or text.

26-160430

Hol dir deinen kostenlosen, vertraulichen Lebenslauf-Check.

oder ziehe deine Datei hierhin.

Similar jobs

Ähnliche Jobs, die dir auch gefallen könnten

AI / ML Engineer
AI / ML Engineer

INSPYR Solutions • Brentwood (TN)

Remote
USD 96.000 - 110.000
Flexible remote work opportunities
Professional development programs
Employee wellbeing culture
Software Engineer II - ML Systems (Remote)
Software Engineer II - ML Systems (Remote)

INSPYR Solutions • California

Hybrid
USD 102.000 - 144.000
Comprehensive medical benefits
Competitive pay
401(k) retirement plan
+1
Artificial Intelligence Machine Learning Solutions Engineer 2
Artificial Intelligence Machine Learning Solutions Engineer 2

INSPYR Solutions • Charlotte (NC)

Hybrid
USD 130.000 - 150.000
Machine Learning Engineer
Machine Learning Engineer

INSPYR Solutions • Sunnyvale (CA)

Hybrid
USD 120.000 - 180.000
AI/ML Quality Engineer — LLM Evaluation & QA Automation
AI/ML Quality Engineer — LLM Evaluation & QA Automation

INSPYR Solutions • Cupertino (CA)

Hybrid
USD 91.000 - 105.000
Remote-friendly schedule
AI / ML & Generative AI Engineer
AI / ML & Generative AI Engineer

INSPYR Solutions • USA

Remote
USD 124.000 - 220.000
Flexible remote work
Continuous learning
Employee wellbeing
Data Operations Program Manager
Data Operations Program Manager

INSPYR Solutions • Seattle (WA)

Vor Ort
USD 76.000 - 103.000
Sr. AI Agentic Engineer (Software Engineer)
Sr. AI Agentic Engineer (Software Engineer)

INSPYR Solutions • Fort Worth (TX)

Vor Ort
USD 120.000 - 180.000
Comprehensive medical benefits
Competitive pay
401(k) retirement plan
+1
Quality Assurance Tech II
Quality Assurance Tech II

INSPYR Solutions • North Reading (MA)

Vor Ort
USD 41.000 - 55.000
Comprehensive medical benefits
Competitive pay
401(k) retirement plan
Software Engineer II
Software Engineer II

INSPYR Solutions • California

Hybrid
USD 102.000 - 144.000
Comprehensive medical benefits
Competitive pay
401(k) retirement plan
+1