AI Benchmarking Research Engineer

Ramp; New York (NY); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Flexible PTO

Centralized home-office equipment

Health and wellness stipend

Travel budget

Weekly coffee stipend

Job summary

Ramp is seeking an ML benchmarking engineer to own Ramp's benchmarking effort. You will turn real finance workflows (expense policy review, invoice coding, month-end close, spreadsheet Q&A) into benchmarks and RL environments, build graders, and tell the team what is actually improving.

You'll collaborate with Ramp Labs to design evals, create versioned benchmark suites, and publish results that guide model development and deployment for finance-related tasks across production systems.

Qualifications

• Hands-on experience building LLM benchmarks or evals used to drive decisions.
• Strong Python and software engineering; eval code must be reproducible and fast.
• Statistical judgment: variance, confidence intervals, sample size, inter-rater agreement.
• Working knowledge of post-training (SFT, RL with verifiable rewards) and how evals feed into it.

Responsibilities

• Build benchmarks from real workflows: agent traces, edge cases, and labels into versioned benchmark suites.
• Own benchmarking: decide what to measure, run suites across models, report results.
• Design graders: rubrics and llm-as-judge, calibrated against human agreement.
• Close the loop with post-training: turn evals into reward functions and RL environments.

Skills

LLM benchmarks

Python

Software engineering

Statistical judgment

Post-training knowledge

SQL & large datasets

Taste for what to measure

Tools

SQL

Job description

Ramp is seeking an ML benchmarking engineer to own Ramp's benchmarking effort. You will turn real finance workflows (expense policy review, invoice coding, month-end close, spreadsheet Q&A) into benchmarks and RL environments, build graders, and tell the team what is actually improving.

You'll collaborate with Ramp Labs to design evals, create versioned benchmark suites, and publish results that guide model development and deployment for finance-related tasks across production systems.