Benchmarking Software Engineer — Remote

Office Hours; United States; On-site; USD 160,000 - 210,000; Full time

Benefits offered by this job

Equity

Medical/dental/vision

401(k)

Wellness stipend

Paid time off

Off-sites

Remote flexibility

Job summary

Office Hours is seeking a Software Engineer to build and run the platform behind our AI model evaluations. You will partner with researchers to prepare benchmark datasets, create repeatable evaluation pipelines, and deliver publishable results.

You will design containerized environments, support model experiments, and develop dashboards and leaderboards to track progress. This role offers remote or hybrid work from SF/NYC and a competitive package.

Qualifications

• 4+ years of professional software engineering experience.
• Strong Python and Docker experience.
• Ability to work with researchers and engineers to translate methodology into systems.

Responsibilities

• Prepare and maintain benchmark datasets and cleaning.
• Build evaluation pipelines across model APIs and agents to ensure reproducibility.
• Create containerized evaluation environments and viewers for tasks and model performance.
• Support model experiments and compare baselines.
• Develop the scoreboard and leaderboards for results across models and tasks.
• Build analysis tools to identify failure modes and track improvements over time.
• Collaborate with researchers and engineers to integrate data and outputs into publications.
• Build lightweight HTML viewers and internal tools for task authoring and data collection.

Skills

Python

Docker

Data quality

Collaboration

Tools

Harbor

Terminal-Bench

Inspect

Job description

Office Hours is seeking a Software Engineer to build and run the platform behind our AI model evaluations. You will partner with researchers to prepare benchmark datasets, create repeatable evaluation pipelines, and deliver publishable results.

You will design containerized environments, support model experiments, and develop dashboards and leaderboards to track progress. This role offers remote or hybrid work from SF/NYC and a competitive package.