AI QA Trainer - Freelance Project

agency; United States; Remote; USD 8,300 - 90,000; Full time

Benefits offered by this job

Remote contract

Flexible hourly rate

Job summary

agency is seeking a contract QA specialist to assess the accuracy, safety, and reliability of language models through structured testing and failure analysis. This remote, contract role focuses on evaluation design, adversarial testing, and quality reporting for AI/ML systems.

You will design test plans, run regression checks, document reproducible failures, and report findings with clear root-cause hypotheses. The hourly pay ranges from $6 to $65, depending on experience and location.

Qualifications

• Strong understanding of AI/ML quality assurance and model evaluation.
• Clear, evidence-based communication and issue reproduction capabilities.
• Degree in CS, data science, computational linguistics, statistics, or related field preferred.
• Experience shipping QA for AI/ML systems and designing rubrics is valuable.
• Python or SQL test automation experience; familiarity with PyTest or evaluation frameworks helpful.
• Knowledge of grounding checks, fairness audits, regression testing, and retrieval-augmented systems beneficial.

Responsibilities

• Test model responses against realistic prompts and scenarios for accuracy and grounding.
• Design test plans, regression suites, scoring rubrics, and pass/fail criteria.
• Evaluate risks such as hallucinations, jailbreaks, bias, unsafe outputs, and retrieval errors.
• Document reproducible failures and propose root-cause hypotheses; report findings clearly.
• Suggest improvements to prompts, safeguards, and evaluation measures like precision and latency.
• Contribute to red-team exercises, test automation, and dashboards tracking quality over time.

Skills

Quality assurance AI/ML

Clear communication

QA for AI/ML systems

Python/SQL test automation

PyTest / evaluation frameworks

Grounding checks & fairness audits

Education

CS/DS/Computational Linguistics degree

Tools

PyTest

LLM evaluation frameworks

Retrieval-augmented systems knowledge

Job description

Role overview

Assess the accuracy, safety, and reliability of language models through structured testing and detailed failure analysis. This contract role spans evaluation design, adversarial testing, regression checks, and quality reporting, helping teams identify weaknesses in model responses and workflows.

Responsibilities

• Test model responses against realistic prompts and scenarios, checking factual accuracy, reasoning, and grounding.
• Design test plans, regression suites, scoring rubrics, and clear pass/fail criteria.
• Evaluate risks such as hallucinations, prompt injection, jailbreaks, bias, unsafe outputs, incorrect tool use, and retrieval errors.
• Document reproducible failures, propose root-cause hypotheses, and report findings clearly.
• Recommend improvements to prompts, safeguards, and evaluation measures such as precision, recall, faithfulness, toxicity, or latency.
• Contribute to red-team exercises, test automation, and dashboards that track quality over time.

Requirements

• Strong understanding of AI or language-model quality assurance, model evaluation, safety testing, or a related discipline.
• Clear, evidence-based communication, including the ability to explain evaluation decisions and reproduce issues.
• A degree in computer science, data science, computational linguistics, statistics, or a related field is preferred.
• Experience shipping QA for AI/ML systems, designing rubrics, or conducting adversarial testing is valuable.
• Python or SQL test automation experience is useful; familiarity with tools such as PyTest or LLM evaluation frameworks is a plus.
• Knowledge of grounding checks, fairness audits, regression testing, and retrieval-augmented systems is beneficial.

Benefits and work setup

• Remote, contract work.
• The listed hourly range is $6–$65; the final rate depends on experience, expertise, and location.
• Contractors supply a secure computer and high-speed internet. Employer-sponsored health insurance and paid time off are not included.