Benchmarking Software Engineer — Remote, Equity

Office-Hours; San Francisco, New York (CA, NY); Hybrid; USD 160,000 - 210,000; Full time

Benefits offered by this job

Competitive salary

Equity

Medical, dental, and vision coverage

401(k)

Monthly wellness stipend

Paid time off and holidays

Annual off-sites

Remote flexibility

Job summary

Office Hours is seeking a Software Engineer to build and run the platform behind our AI model evaluations. You will work with researchers to prepare benchmark datasets, create reproducible evaluation pipelines, and turn results into publishable outputs.

You will implement containerized environments, support experiments on open-source LLMs, and develop dashboards and leaderboards to track performance across models and tasks.

Qualifications

• 4+ years of professional software engineering experience.
• Strong Python programming skills.
• Experience with containerization and reproducible environments.

Responsibilities

• Prepare and maintain benchmark datasets with clean, runnable formats.
• Build and maintain evaluation pipelines across model APIs.
• Create containerized evaluation environments.
• Support model experiments and fine-tuning open-source LLMs.
• Develop scoreboard, leaderboards, and analysis tools.
• Collaborate with researchers and engineers on data integration.
• Build data creation and review tooling for annotation tasks.

Skills

Python

Data handling

Containerization

Collaboration

Tools

Docker

GitHub

Hugging Face

PyTorch

React

Job description

Office Hours is seeking a Software Engineer to build and run the platform behind our AI model evaluations. You will work with researchers to prepare benchmark datasets, create reproducible evaluation pipelines, and turn results into publishable outputs.

You will implement containerized environments, support experiments on open-source LLMs, and develop dashboards and leaderboards to track performance across models and tasks.