Benchmarking Engineer for AI in Finance

Ramp; San Francisco (CA); On-site; USD 150,000 - 200,000; Full time

Benefits offered by this job

Flexible PTO

Home-office equipment ordering

Health and wellness stipend

Relocation expense coverage to NYC or

Weekly coffee stipend

Job summary

Ramp Labs is seeking a benchmarks-focused engineer to turn real finance work (expense policy review, invoice coding, month‑end close) into robust benchmarks and RL environments. You’ll build graders, design evaluation frameworks, and help the team improve models used to move money and close books.

You’ll own the benchmarking pipeline, turn evals into actionable rewards, and publish benchmarks and analyses that inform product decisions.

Qualifications

• Hands-on experience building LLM benchmarks or evals that people used to make real decisions.
• Strong Python and software engineering; your eval code is reproducible and fast.
• Statistical judgment: variance, confidence intervals, sample size, inter‑rater agreement.
• Working knowledge of post‑training (SFT, RL with verifiable rewards) and how evals feed into it.
• Comfort with SQL and large datasets; you’ll work directly from production traces.
• Taste for what is worth measuring.

Responsibilities

• Build benchmarks from real workflows: convert agent traces and expert labels into versioned benchmark suites for long‑horizon agents.
• Own the benchmarking effort: decide metrics, run suites across models, report results for team decisions.
• Design graders: rubrics, programmatic checks and llm‑as‑judge, calibrated to human agreement.
• Close the loop with post‑training: turn evals into reward functions and RL environments, catch reward hacking.
• Publish external benchmarks and write‑ups on model performance in real finance work.

Skills

LLM benchmarks

Python

Statistics

Post-training

SQL

Tools

LM evaluation harness

HELM

Job description

Ramp Labs is seeking a benchmarks-focused engineer to turn real finance work (expense policy review, invoice coding, month‑end close) into robust benchmarks and RL environments. You’ll build graders, design evaluation frameworks, and help the team improve models used to move money and close books.

You’ll own the benchmarking pipeline, turn evals into actionable rewards, and publish benchmarks and analyses that inform product decisions.