Benchmarking Software Engineer (Remote)

Office Hours; United States; Hybrid; USD 160,000 - 210,000; Full time

Benefits offered by this job

Competitive salary and equity

Medical, dental, and vision coverage

401(k)

Monthly wellness and fitness stipend

Paid time off and company holidays

Annual company off-sites

Remote flexibility

Job summary

Office Hours is seeking a Software Engineer to build and run the platform behind our AI model evaluations. You’ll prepare benchmark datasets, construct reproducible evaluation pipelines, and create containerized environments to run models and tools.

Work closely with researchers to translate methodology into scalable systems. You will help fine-tune small models, develop leaderboards, and craft analysis tools to identify failure modes and track improvements over time, with a flexible remote or

Qualifications

• 4+ years of professional experience building and maintaining complex systems.
• Strong Python programming skills and code quality discipline.
• Experience preparing, cleaning, and maintaining benchmark datasets to ensure comparability.
• Ability to work with researchers to translate methodology into production-ready systems.

Responsibilities

• Prepare and maintain benchmark datasets for evaluations.
• Build and maintain evaluation pipelines for consistent results.
• Create containerized evaluation environments and viewers for tasks.
• Support fine-tuning and comparison of models and baselines.
• Develop the scoreboard and leaderboard views for multi-level results.
• Build tools to identify failure modes and track improvements over time.
• Collaborate with researchers and engineers to ensure accurate, integrated outputs.
• Build lightweight tooling for data creation and review.

Skills

Python

Data rigor

Docker

Collaborative

Tools

Docker

Job description

Office Hours is seeking a Software Engineer to build and run the platform behind our AI model evaluations. You’ll prepare benchmark datasets, construct reproducible evaluation pipelines, and create containerized environments to run models and tools.

Work closely with researchers to translate methodology into scalable systems. You will help fine-tune small models, develop leaderboards, and craft analysis tools to identify failure modes and track improvements over time, with a flexible remote or