Remote AI Evaluation Infrastructure Engineer

Block, Inc.; Northern (KY); Hybrid; USD 264,000 - 395,000; Full time

Benefits offered by this job

Remote work

Medical insurance

Flexible time off

Retirement savings plan

Family planning options

Job summary

Block, Inc. is hiring to build the infrastructure that enables high-quality AI evaluation at scale. You will empower teams to determine if model changes improve performance, ensuring online outcomes align with offline metrics.

You will work on offline/online eval systems, dashboards, and tooling that speed up development, enable trustworthy experimentation, and accelerate AI product iteration across the company.

Qualifications

• Experience building production platforms or infrastructure.
• Strong statistics with confidence intervals, variance, power, and multiple comparisons.
• Judgment to distinguish meaningful results from noise.
• Experience evaluating LLM/ML systems or deep systems engineering with AI focus.
• Product instincts for internal tools like leaderboards and workflows.

Responsibilities

• Build an execution engine that scores candidate versions against task sets in minutes.
• Create task set tooling that samples from production logs and validates tasks.
• Build grader infrastructure across ground truth, rubrics, and LLM-as-judge approaches.
• Develop tooling to calibrate judges and monitor drift over time.
• Build leaderboards and reporting systems with sample size and confidence intervals.
• Support in-product side-by-side serving and implicit signal extraction.
• Develop loops comparing offline scores with online outcomes to improve eval sets.
• Partner with product, engineering, data, and ML teams to make evaluation workflows fast and trustworthy.

Skills

Distributed systems

Data pipelines

Production logs

Statistical literacy

LLM/ML evaluation

Cross-functional collaboration

Observability

Experimentation

Job description

Block, Inc. is hiring to build the infrastructure that enables high-quality AI evaluation at scale. You will empower teams to determine if model changes improve performance, ensuring online outcomes align with offline metrics.

You will work on offline/online eval systems, dashboards, and tooling that speed up development, enable trustworthy experimentation, and accelerate AI product iteration across the company.