LLM Evaluation Researcher/Benchmarking Engineer

House of Scribes; Seattle (WA); Vor Ort; USD 21.000 - 23.000; Vollzeit

Zusammenfassung

House of Scribes is seeking an experienced Python/LLM evaluation engineer to implement and stress-test a 20-dimension narrative assessment framework. The role focuses on building a reproducible Python-based evaluation harness and delivering transparent results within a fixed budget.

You will prepare 16 global benchmark works, evaluate eight additional controlled works, and perform blind, repeatability and diagnostic tests, while tracking costs and token usage to provide a concise technical

Qualifikationen

• Strong Python and LLM API experience.
• Experience with LLM evaluation, benchmarking or AI quality assurance.
• Structured outputs, automated testing, source-document parsing and reliable logging.
• Ability to work independently within a fixed budget.

Aufgaben

• Build a reproducible Python-based LLM evaluation harness.
• Prepare and test 16 global benchmark works.
• Evaluate eight additional controlled works and their test variations.
• Run blind, repeatability and diagnostic sensitivity tests.
• Track ratings, source evidence, model consistency, token usage and API costs.
• Deliver working source code, JSON/CSV results and a concise technical report.

Kenntnisse

Python

LLM evaluation

Automated testing

Logging & traceability

Jobbeschreibung

Budget: $2000 fixed

I'm developing a proprietary AI-assisted story evaluation system for film, television and literature.

I need an experienced Python/LLM evaluation engineer to implement and stress-test an existing 20-dimension narrative assessment framework.

The framework, evaluation protocol and initial global benchmark manifest are already prepared.

Responsibilities

• Build a reproducible Python-based LLM evaluation harness.
• Prepare and test 16 global benchmark works.
• Evaluate eight additional controlled works and their supplied test variations.
• Run blind, repeatability and diagnostic sensitivity tests.
• Track ratings, source evidence, model consistency, token usage and API costs.
• Deliver working source code, JSON/CSV results and a concise technical report.

Requirements

• Strong Python and LLM API experience.
• Experience with LLM evaluation, benchmarking or AI quality assurance.
• Structured outputs, automated testing, source-document parsing and reliable logging.
• Ability to work independently within a fixed budget.

This is not a website development, chatbot, creative writing or model-training project.

The objective is to test an existing diagnostic system rigorously, identify weaknesses and produce reproducible results.