AI QA Trainer: LLM Evaluation & Safety Expert

Agency; United States; Remote; USD 120,000 - 160,000; Full time

Job summary

Agency is seeking AI QA experts to harden model reasoning and reliability for enterprise-grade, safe, and dependable language models. You will design evaluation prompts, build rubrics, and run regression suites to tighten performance and safety. Collaboration across teams will shape guardrails, prompts, and dashboards.

The role requires hands-on QA for ML systems, strong Python/SQL automation, and clear bug reporting. Experience with eval tooling and adversarial testing is a plus.

Qualifications

• Experience in model evaluation, LLM safety, and prompt robustness.
• Expertise in hallucination detection and factual consistency.
• Knowledge of prompt-injection and jailbreak resistance.
• Ability to perform bias/fairness audits and grounding verification.
• Proficiency in test automation using Python or SQL.
• Strong metacognitive communication skills for high-signal bug reporting.

Responsibilities

• Converse with models on real-world scenarios and evaluation prompts.
• Verify factual accuracy and logical soundness.
• Design and run test plans and regression suites.
• Build clear rubrics and pass/fail criteria.
• Capture reproducible error traces with root-cause hypotheses.
• Suggest improvements to prompt engineering, guardrails, and evaluation metrics.
• Partner on adversarial red-teaming, automation, and quality dashboarding.

Skills

Model evaluation

LLM safety

Prompt robustness

Hallucination detection

Bias/fairness audits

Grounding verification

Test automation

Python

SQL

Bug reporting

Education

Bachelor’s/Master’s/PhD in CS/DS/Computational Linguistics/Statistics

Experience shipping QA for ML/AI systems

Tools

PyTest

OpenAI Evals

RAG evaluators

Weights & Biases (W&B)

Job description

Agency is seeking AI QA experts to harden model reasoning and reliability for enterprise-grade, safe, and dependable language models. You will design evaluation prompts, build rubrics, and run regression suites to tighten performance and safety. Collaboration across teams will shape guardrails, prompts, and dashboards.

The role requires hands-on QA for ML systems, strong Python/SQL automation, and clear bug reporting. Experience with eval tooling and adversarial testing is a plus.