Member of Technical Staff - ML Evaluation & Benchmarking

Arena Physica; New York (NY); On-site; USD 175,000 - 250,000; Full time

Benefits offered by this job

Premium health benefits

401(k)

Unlimited PTO

Lunch via Sharebite

Relocation support

Job summary

Arena Physica is hiring a Member of Technical Staff for Evaluations to lead the design, verification, and automation of evaluation pipelines for hardware-focused AI systems. You will craft analyses, calibrate scoring, and report findings to CTO and partners.

The role spans infrastructure, statistics, and domain knowledge, with collaboration across electrical engineering and product teams. This is a high-impact role based in the New York area.

Qualifications

• Strong Python skills with experience building research or production infrastructure.
• Experience designing evaluations, benchmarks, or metrics for ML systems.
• Hands-on experience with LLM agents and analysis of agent trajectories.
• Experience with experimental design and statistics.

Responsibilities

• Own design and reliability of evaluation harnesses and automated pipelines.
• Design ablations that isolate the effect of model changes on performance.
• Analyze agent trajectories to identify performance clusters and root causes.
• Collaborate to hand-craft synthetic datasets that test capabilities.
• Design and calibrate LLM judges and scoring systems.

Skills

Python

Eval design

LLM agents

Experimental design

Communication

Automation bias

EE background

LLM judge

Job description

Arena Physica is hiring a Member of Technical Staff for Evaluations to lead the design, verification, and automation of evaluation pipelines for hardware-focused AI systems. You will craft analyses, calibrate scoring, and report findings to CTO and partners.

The role spans infrastructure, statistics, and domain knowledge, with collaboration across electrical engineering and product teams. This is a high-impact role based in the New York area.