AI QA Trainer - LLM Evaluation - Freelance Project

remote work; United States; Teletrabalho; USD 83 000 - 138 000; Tempo integral

Resumo da oferta

Meridial is seeking a contractor to design test plans and evaluate large language models for safety, reliability, and prompt robustness. You will own test plans, execute evaluations, document failure modes, and suggest prompt and guardrail improvements.

Responsibilities include adversarial testing, bias audits, and building reproducible error traces; you will automate workflows with Python/SQL and help track quality deltas via dashboards. Strong experience in ML QA and LLM tooling is required.

Qualificações

• Experience in ML/AI QA, safety/red-team activities, and test automation frameworks.
• Hands-on work with LLM evaluation tooling (e.g., OpenAI Evals, RAG evaluators).
• Strong ability to design evaluation rubrics and conduct regression testing at scale.
• Proficiency in Python and SQL for automation and data analysis.
• Clear, metacognitive communication and ability to document findings and root causes.
• Advanced degree or equivalent in CS, data science, computational linguistics, or related field preferred.

Responsabilidades

• Converse with models on real-world scenarios and evaluation prompts to assess factual accuracy and logical soundness.
• Design and run test plans and regression suites for LLM evaluation.
• Build rubrics and pass/fail criteria; capture reproducible error traces with root-cause hypotheses.
• Adversarial red-teaming, bias/fairness audits, and grounding verification tasks.
• Automate workflows with Python/SQL and contribute to dashboards tracking quality deltas.
• Collaborate on prompt engineering improvements and evaluation metrics such as precision/recall and latency SLOs.

Conhecimentos

ML QA

Safety red-team

Test automation

Python

SQL

LLM evaluation tooling

OpenAI Evals

RAG evaluators

Formação académica

Advanced degree

Ferramentas

OpenAI Evals

RAG evaluators

Descrição da oferta de emprego

About the role

Meridial is building AI QA processes for evaluating large language models, focusing on safety, reliability, and prompt robustness. This role owns designing test plans, executing evaluations, documenting failure modes, and suggesting improvements to prompts, guardrails, and evaluation metrics as a contractor.

What you'll do

• Converse with models on real-world scenarios and evaluation prompts to assess factual accuracy and logical soundness
• Design and run test plans and regression suites for LLM evaluation
• Build rubrics and pass/fail criteria; capture reproducible error traces with root-cause hypotheses
• Adversarial red-teaming, bias/fairness audits, and grounding verification tasks
• Automate workflows with Python/SQL and contribute to dashboards tracking quality deltas
• Collaborate on prompt engineering improvements and evaluation metrics such as precision/recall and latency SLOs

What they're looking for

• Experience in ML/AI QA, safety/red-team activities, and test automation frameworks
• Hands-on work with LLM evaluation tooling (e.g., OpenAI Evals, RAG evaluators)
• Strong ability to design evaluation rubrics and conduct regression testing at scale
• Proficiency in Python and SQL for automation and data analysis
• Clear, metacognitive communication and ability to document findings and root causes
• Advanced degree or equivalent in CS, data science, computational linguistics, or related field preferred

For applicants in Nigeria

• The listing states worldwide remote eligibility, so Nigeria is included as a potential location
• Contract work with pay ranges disclosed; exact rate determined after evaluation of experience and geographic location
• The listing does not specify a timezone requirement; confirm timezone expectations before interviewing
• The listing notes contractor status and does not provide company-sponsored benefits such as health insurance or PTO
• Payment currency is USD as indicated by the pay range; confirm currency details with the employer