Remote AI QA Trainer: LLM Evaluation & Reliability

Agency; United States; Remote; USD 8,300 - 90,000; Full time

Job summary

Agency seeks AI QA trainers to rigorously evaluate large language models across multilingual and domain-specific contexts. You will design tests, run regression suites, and document failure modes to raise the bar for model reasoning, reliability, and safety.

You’ll work on adversarial red-teaming, automation with Python/SQL, and dashboards to track quality deltas. A strong background in ML/AI QA and tooling signals fit.

Qualifications

• A bachelor’s, master’s, or PhD in CS/DS/CL/ statistics is ideal.
• Shipped QA for ML/AI systems and safety/red-team experience.
• Experience with test automation frameworks (e.g., PyTest).
• Hands-on with LLM eval tooling (OpenAI Evals, RAG evaluators, W&B).

Responsibilities

• Converse with the model on real-world scenarios and evaluation prompts, verify factual accuracy and logical soundness.
• Design and run test plans and regression suites, build clear rubrics and pass/fail criteria.
• Capture reproducible error traces with root-cause hypotheses, and suggest improvements to prompt engineering, guardrails, and evaluation metrics.
• Partner on adversarial red-teaming, automation (Python/SQL), and dashboarding to track quality deltas over time.

Skills

Model evaluation

LLM safety

Prompt robustness

Data quality assurance

Multilingual testing

Grounding verification

Compliance checks

Education

CS/ML degree

PhD ideal

Tools

Python

SQL

Job description

Agency seeks AI QA trainers to rigorously evaluate large language models across multilingual and domain-specific contexts. You will design tests, run regression suites, and document failure modes to raise the bar for model reasoning, reliability, and safety.

You’ll work on adversarial red-teaming, automation with Python/SQL, and dashboards to track quality deltas. A strong background in ML/AI QA and tooling signals fit.