Remote AI Measurement Architect — LLM Evaluation

Remote Jobs; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

Broadcastwell LLC is hiring for a remote, full-time role in the United States. The position focuses on reliability and reproducibility of measurement for AI engine evaluations, with ownership of scoring rules and traceable data workflows.

You will design how measurement runs are recorded, prepare datasets for each volume, and decide when results are publishable, documenting any limitations. Flexible hours align with US time zones.

Qualifications

• 5+ years building production systems.
• Proficient in Python, system design, and data pipelines.
• Hands-on experience evaluating large language model outputs.
• A reproducible analysis or evaluation that others can rerun.

Responsibilities

• Own the scoring rules and document them on the methodology page.
• Design measurement run recording so citations trace to answers.
• Turn research requirements into repeatable measurement designs.
• Prepare datasets and analyses for published research volumes.
• Decide when results are solid to publish and note limitations.
• Review client measurements before findings are shared.

Skills

Python

System design

Data pipelines

LLM evaluation

Reproducible analysis

Job description

Broadcastwell LLC is hiring for a remote, full-time role in the United States. The position focuses on reliability and reproducibility of measurement for AI engine evaluations, with ownership of scoring rules and traceable data workflows.

You will design how measurement runs are recorded, prepare datasets for each volume, and decide when results are publishable, documenting any limitations. Flexible hours align with US time zones.