AI Evaluation Platform Engineer

Block; San Francisco (CA); Remote; USD 264,000 - 395,000; Full time

Job summary

Block in San Francisco is hiring an engineer to build the infrastructure for high‑quality AI evaluation at scale. You will help teams determine if model changes improve outcomes, and if offline scores predict online behavior.

You will build systems that run fast, measure confidence, and ensure trustworthy evaluation across product, data, and engineering teams. The role emphasizes reliability, observability, and collaboration to ship better AI products.

Qualifications

• Experience building production platforms or infrastructure, including distributed batch execution, data pipelines, or systems that process production logs.
• Strong statistical literacy, including confidence intervals, variance, power, and multiple comparisons.
• The judgment to identify when a result is meaningful and when it is noise.
• Experience evaluating LLM or ML systems, or deep systems engineering with a strong interest in AI evaluation.
• Product instinct for internal tools; leaderboards, annotation tools, and workflows must be used by teams.
• A bias toward building reliable, observable systems that other engineers can trust.
• Strong collaboration skills across ambiguous product, data, and engineering problems.

Responsibilities

• Build an execution engine that can score candidate versions against task sets in minutes, not hours.
• Create task set tooling that samples from production logs and validates tasks before admission into an evaluation set.
• Build grader infrastructure across ground truth checks, rubrics, and LLM-as-judge approaches.
• Develop tooling that helps human reviewers calibrate judges, measure judge-to-human agreement, and monitor drift over time.
• Build leaderboards and reporting systems with sample size, confidence intervals, and run-to-run variance.
• Support in-product side-by-side serving, feedback capture, and implicit signal extraction from real conversations.
• Build the loop that compares offline scores with online outcomes and improves eval sets.
• Partner with product, engineering, data, and ML teams to make evaluation workflows fast and trustworthy.

Skills

Production platforms

Statistics

Signal interpretation

LLM/ML evaluation

Internal tools mindset

Reliable systems

Cross-team collaboration

Job description

Block in San Francisco is hiring an engineer to build the infrastructure for high‑quality AI evaluation at scale. You will help teams determine if model changes improve outcomes, and if offline scores predict online behavior.

You will build systems that run fast, measure confidence, and ensure trustworthy evaluation across product, data, and engineering teams. The role emphasizes reliability, observability, and collaboration to ship better AI products.