AI Evaluation Platform Engineer (Remote)

Block; Sacramento (CA); Remote; USD 264,000 - 395,000; Full time

Benefits offered by this job

Remote work

Medical insurance

Flexible time off

Retirement savings plans

Family planning

Job summary

Block is hiring an engineer to build infrastructure and tooling for high-quality AI evaluation at scale. You will create systems to compare candidate versions, measure accuracy, and ensure offline and online results align.

The team collaborates with product, data, and ML groups to deliver fast, trustworthy evaluation workflows. This role emphasizes reliable, observable systems, rapid iteration, and measurable improvements in evaluation quality that inform product decisions across Block's AI

Qualifications

• Experience building production platforms or infrastructure, including distributed batch execution, data pipelines, or systems that process production logs.
• Strong statistical literacy, including confidence intervals, variance, power, and multiple comparisons.
• Judgment to identify when a result is meaningful and when it is noise.
• Experience evaluating LLM or ML systems, or deep systems engineering with a strong interest in AI evaluation.
• Product instinct for internal tools. You understand that leaderboards, annotation tools, and workflows only matter if teams actually use them.
• A bias toward building reliable, observable systems that other engineers can trust.
• Strong collaboration skills across ambiguous product, data, and engineering problems.
• In your first year: product teams can stand up credible evals for new AI surfaces in days, and CI gates run quickly.

Responsibilities

• Build an execution engine that can score candidate versions against task sets in minutes, not hours.
• Create task set tooling that samples from production logs and validates tasks before they are admitted into an evaluation set.
• Build grader infrastructure across ground truth checks, rubrics, and LLM-as-judge approaches.
• Develop tooling that helps human reviewers calibrate judges, measure judge-to-human agreement, and monitor drift over time.
• Build leaderboards and reporting systems that include sample size, confidence intervals, and run-to-run variance.

Skills

Distributed batch execution

Data pipelines

Production logs

LLM/ML evaluation

Observability

Cross-functional collaboration

Tools

Distributed systems tooling

Job description

Block is hiring an engineer to build infrastructure and tooling for high-quality AI evaluation at scale. You will create systems to compare candidate versions, measure accuracy, and ensure offline and online results align.

The team collaborates with product, data, and ML groups to deliver fast, trustworthy evaluation workflows. This role emphasizes reliable, observable systems, rapid iteration, and measurable improvements in evaluation quality that inform product decisions across Block's AI