LLM Evaluation QA Engineer - Remote Contract

remote work; United States; Remote; USD 83.000 - 138.000; Vollzeit

Zusammenfassung

Meridial is seeking a contractor to design test plans and evaluate large language models for safety, reliability, and prompt robustness. You will own test plans, execute evaluations, document failure modes, and suggest prompt and guardrail improvements.

Responsibilities include adversarial testing, bias audits, and building reproducible error traces; you will automate workflows with Python/SQL and help track quality deltas via dashboards. Strong experience in ML QA and LLM tooling is required.

Qualifikationen

• Experience in ML/AI QA, safety/red-team activities, and test automation frameworks.
• Hands-on work with LLM evaluation tooling (e.g., OpenAI Evals, RAG evaluators).
• Strong ability to design evaluation rubrics and conduct regression testing at scale.
• Proficiency in Python and SQL for automation and data analysis.
• Clear, metacognitive communication and ability to document findings and root causes.
• Advanced degree or equivalent in CS, data science, computational linguistics, or related field preferred.

Aufgaben

• Converse with models on real-world scenarios and evaluation prompts to assess factual accuracy and logical soundness.
• Design and run test plans and regression suites for LLM evaluation.
• Build rubrics and pass/fail criteria; capture reproducible error traces with root-cause hypotheses.
• Adversarial red-teaming, bias/fairness audits, and grounding verification tasks.
• Automate workflows with Python/SQL and contribute to dashboards tracking quality deltas.
• Collaborate on prompt engineering improvements and evaluation metrics such as precision/recall and latency SLOs.

Kenntnisse

ML QA

Safety red-team

Test automation

Python

SQL

LLM evaluation tooling

OpenAI Evals

RAG evaluators

Ausbildung

Advanced degree

Tools

OpenAI Evals

RAG evaluators

Jobbeschreibung

Meridial is seeking a contractor to design test plans and evaluate large language models for safety, reliability, and prompt robustness. You will own test plans, execute evaluations, document failure modes, and suggest prompt and guardrail improvements.

Responsibilities include adversarial testing, bias audits, and building reproducible error traces; you will automate workflows with Python/SQL and help track quality deltas via dashboards. Strong experience in ML QA and LLM tooling is required.