LLM Evaluation Engineer — Benchmarking & Diagnostics

House of Scribes; Seattle (WA); On-site; USD 21,000 - 23,000; Full time

Job summary

House of Scribes is seeking an experienced Python/LLM evaluation engineer to implement and stress-test a 20-dimension narrative assessment framework. The role focuses on building a reproducible Python-based evaluation harness and delivering transparent results within a fixed budget.

You will prepare 16 global benchmark works, evaluate eight additional controlled works, and perform blind, repeatability and diagnostic tests, while tracking costs and token usage to provide a concise technical

Qualifications

• Strong Python and LLM API experience.
• Experience with LLM evaluation, benchmarking or AI quality assurance.
• Structured outputs, automated testing, source-document parsing and reliable logging.
• Ability to work independently within a fixed budget.

Responsibilities

• Build a reproducible Python-based LLM evaluation harness.
• Prepare and test 16 global benchmark works.
• Evaluate eight additional controlled works and their test variations.
• Run blind, repeatability and diagnostic sensitivity tests.
• Track ratings, source evidence, model consistency, token usage and API costs.
• Deliver working source code, JSON/CSV results and a concise technical report.

Skills

Python

LLM evaluation

Automated testing

Logging & traceability

Job description

House of Scribes is seeking an experienced Python/LLM evaluation engineer to implement and stress-test a 20-dimension narrative assessment framework. The role focuses on building a reproducible Python-based evaluation harness and delivering transparent results within a fixed budget.

You will prepare 16 global benchmark works, evaluate eight additional controlled works, and perform blind, repeatability and diagnostic tests, while tracking costs and token usage to provide a concise technical