AI Evaluation Platform Engineer

Block; United States; Remote; USD 264,000 - 395,000; Full time

Job summary

Block is hiring an engineer to build the infrastructure and tooling for high-quality AI evaluation at scale. You will help determine whether model or product changes are meaningful, and ensure offline evaluation predicts real user outcomes.

You will develop an execution engine, task-set tooling, grader infrastructure, and dashboards to empower product teams to ship better AI products quickly and reliably.

Qualifications

• Experience building production platforms or infrastructure.
• Strong statistical literacy, including CI, variance, power, and multiple comparisons.
• Judgment to distinguish meaningful results from noise.
• Experience evaluating LLM/ML systems or deep systems engineering with AI evaluation.
• Product instinct for internal tools and usability by teams.
• Bias toward building reliable, observable systems engineers can trust.
• Strong collaboration across ambiguous product, data, and engineering problems.

Responsibilities

• Build an execution engine that can score candidate versions quickly.
• Create task-set tooling sampling from production logs and validating tasks before admission.
• Build grader infrastructure across ground-truth checks, rubrics, and LLM-as-judge approaches.
• Develop tooling to calibrate judges, measure judge-to-human agreement, and monitor drift.
• Build leaderboards and reporting with sample size, confidence intervals, and run-to-run variance.
• Support in-product side-by-side serving, feedback capture, and implicit signal extraction.
• Build the loop that aligns offline scores with online outcomes and improves them.
• Partner with product, engineering, data, and ML teams to speed up evaluation workflows.

Skills

Distributed systems

Data pipelines

Production logs

Statistical literacy

LLM/ML evaluation

Collaboration

Observability

Job description

Block is hiring an engineer to build the infrastructure and tooling for high-quality AI evaluation at scale. You will help determine whether model or product changes are meaningful, and ensure offline evaluation predicts real user outcomes.

You will develop an execution engine, task-set tooling, grader infrastructure, and dashboards to empower product teams to ship better AI products quickly and reliably.