AI QA Trainer – LLM Evaluation

Invisible Technologies; United States; Remote; USD 8,300 - 90,000; Part time

Job summary

Invisible Technologies Inc. seeks an AI QA trainer to harden model reasoning and reliability across multilingual and domain-specific testing.

You will design test plans, build rubrics, and document failure modes to raise standards for safety, bias checks, and prompt robustness. As a contractor, you will work remotely with flexible hours, collaborating with researchers and engineers to evaluate LLM outputs, identify edge cases, and propose improvements to prompts, guardrails, and evaluation

Qualifications

• For ideal fit, advanced degrees or certifications in ML/AI testing, safety red-teaming, and test automation.
• Hands-on experience with LLM eval tooling (OpenAI Evals, RAG evaluators, W&B) signals fit.
• Shipped QA for ML/AI systems and familiarity with evaluation rubrics.

Responsibilities

• Converse with the model on real-world scenarios and evaluation prompts.
• Verify factual accuracy and logical soundness of outputs.
• Design and run test plans and regression suites.
• Build clear rubrics and pass/fail criteria for evaluations.
• Capture reproducible error traces with root-cause hypotheses.
• Suggest improvements to prompt engineering, guardrails, and metrics.
• Partner on adversarial red-teaming, automation (Python/SQL), and dashboards.

Skills

Model evaluation

LLM safety

Prompt robustness

Data quality assurance

Multilingual testing

Domain-specific testing

Grounding verification

Compliance readiness checks

Hallucination detection

Factual consistency

Prompt-injection resistance

Jailbreak resistance

Bias/fairness audits

Chain-of-reasoning reliability

Tool-use correctness

Retrieval-augmentation fidelity

End-to-end workflow validation

Adversarial red-teaming

Python/SQL automation

Education

Bachelor's degree (CS/Data Science/Computational Linguistics/Statistics)

Master's or PhD in related field

Tools

OpenAI Evals

RAG evaluators

Weights & Biases (W&B)

PyTest

Python

SQL

Job description

Are you an AI QA expert eager to shape the future of AI? Large-scale language models are evolving from clever chatbots into enterprise-grade platforms. With rigorous evaluation data, tomorrow’s AI can democratize world-class education, keep pace with cutting-edge research, and streamline workflows for teams everywhere. That quality begins with you—we need your expertise to harden model reasoning and reliability.

We’re looking for AI QA trainers who live and breathe model evaluation, LLM safety, prompt robustness, data quality assurance, multilingual and domain-specific testing, grounding verification, and compliance/readiness checks. You’ll challenge advanced language models on tasks like hallucination detection, factual consistency, prompt-injection and jailbreak resistance, bias/fairness audits, chain-of-reasoning reliability, tool-use correctness, retrieval-augmentation fidelity, and end-to-end workflow validation—documenting every failure mode so we can raise the bar.

On a typical day, you will converse with the model on real-world scenarios and evaluation prompts, verify factual accuracy and logical soundness, design and run test plans and regression suites, build clear rubrics and pass/fail criteria, capture reproducible error traces with root-cause hypotheses, and suggest improvements to prompt engineering, guardrails, and evaluation metrics (e.g., precision/recall, faithfulness, toxicity, and latency SLOs). You’ll also partner on adversarial red-teaming, automation (Python/SQL), and dashboarding to track quality deltas over time.

A bachelor’s, master’s, or PhD in computer science, data science, computational linguistics, statistics, or a related field is ideal; shipped QA for ML/AI systems, safety/red-team experience, test automation frameworks (e.g., PyTest), and hands-on work with LLM eval tooling (e.g., OpenAI Evals, RAG evaluators, W&B) signal fit. Skills that stand out include: evaluation rubric design, adversarial testing/red-teaming, regression testing at scale, bias/fairness auditing, grounding verification, prompt and system-prompt engineering, test automation (Python/SQL), and high-signal bug reporting. Clear, metacognitive communication—"showing your work"—is essential.

Ready to turn your QA expertise into the quality backbone for tomorrow’s AI?

We offer a pay range of $6-to- $65 per hour, with the exact rate determined after evaluating your experience, expertise, and geographic location. Final offer amounts may vary from the pay range listed above. As a contractor you’ll supply a secure computer and high-speed internet; company-sponsored benefits such as health insurance and PTO do not apply.

Employment type: Contract

Workplace type: Remote

Seniority level: Mid-Senior Level

Originally posted on Himalayas