Remote AI QA Architect & Evaluation Lead

Meridial

United States

Remote

USD 8,300 - 90,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Aitrainergigs is seeking an AI QA expert to shape the future of AI evaluation. You will work on model evaluation, safety checks, prompt robustness, data quality, multilingual testing, and grounding verification for enterprise-grade AI platforms.

You will design test plans, run regression suites, and document failures with clear root-cause analyses, contributing to improvements in prompt engineering and evaluation metrics.

Qualifications

  • Experience evaluating language models and ensuring safety, reliability, and prompt robustness.
  • Familiar with QA for ML/AI systems and safety/red-team concepts.
  • Ability to design test plans, rubrics, and regression suites for LLM evaluation.

Responsibilities

  • Converse with models on real-world scenarios and evaluation prompts to verify accuracy and logic.
  • Design and run test plans, build rubrics, and define pass/fail criteria.
  • Capture reproducible error traces with root-cause hypotheses and propose improvements.
  • Collaborate on prompt engineering, guardrails, and evaluation metrics such as precision/recall and fidelity.
  • Engage in adversarial red-teaming, automation (Python/SQL), and dashboarding to track quality deltas.

Skills

AI QA
Evaluation rubric design
Adversarial testing
Regression testing
Multilingual testing

Education

Bachelor's degree in CS / DS / Computational linguistics
Shipped QA for ML/AI systems

Tools

PyTest
OpenAI Evals
RAG evaluators
Weights and Biases (W&B)
Python
SQL

Job description

Aitrainergigs is seeking an AI QA expert to shape the future of AI evaluation. You will work on model evaluation, safety checks, prompt robustness, data quality, multilingual testing, and grounding verification for enterprise-grade AI platforms.

You will design test plans, run regression suites, and document failures with clear root-cause analyses, contributing to improvements in prompt engineering and evaluation metrics.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Remote AI QA Trainer: Evaluation & Safety Lead
Remote AI QA Trainer: Evaluation & Safety Lead

Meridial • United States

Remote
USD 8,300 - 90,000
AI QA Trainer
AI QA Trainer

Meridial • United States

Remote
USD 8,300 - 90,000
Remote QA Engineer - AI Data Quality & Evaluation
Remote QA Engineer - AI Data Quality & Evaluation

YO AI Labs • New York (NY)

Remote
USD 34,000 - 69,000
Remote AI QA Trainer & Evaluation Expert
Remote AI QA Trainer & Evaluation Expert

Invisible Technologies Inc. • United States

Remote
USD 8,300 - 90,000
Remote QA Engineer — AI Training & Evaluation
Remote QA Engineer — AI Training & Evaluation

Xperteez Technology Pvt Ltd • United States

Remote
USD 34,000 - 69,000
AI QA Trainer - LLM Evaluation - Freelance Project
AI QA Trainer - LLM Evaluation - Freelance Project

Agency • United States

Remote
USD 8,300 - 90,000
AI QA Trainer - LLM Evaluation - Freelance Project
AI QA Trainer - LLM Evaluation - Freelance Project

Meridial • United States

Remote
USD 8,265 - 89,544
Secure computer and high-speed internet required
Remote AI QA Trainer: LLM Evaluation & Reliability
Remote AI QA Trainer: LLM Evaluation & Reliability

Agency • United States

Remote
USD 8,300 - 90,000
Remote AI QA Engineer — Data Annotation & Evaluation
Remote AI QA Engineer — Data Annotation & Evaluation

YO AI Labs • New York (NY)

Remote
USD 55,000 - 83,000
AI QA Trainer - LLM Evaluation
AI QA Trainer - LLM Evaluation

Meridial • United States

Remote
USD 8,300 - 90,000