AI Benchmarking Engineer for Finance

Ramp Corp.; New York; Sur place; USD 150 000 - 210 000; Plein temps

Avantages offerts par ce poste

Flexible PTO

Centralized home-office equipment

Health and wellness stipend

Budget for intra-office travel

Weekly coffee stipend

Relocation expense coverage to NYC or

Temporary relocation support to NYC or

Résumé du poste

Ramp Labs is seeking a Benchmark Engineer to turn real finance workflows into robust benchmarks and RL environments. You will design graders, convert eval outcomes into reward signals, and publish findings that guide model improvements used across Ramp’s finance tooling.

In this role, you will own the benchmarking lifecycle, run cross-model comparisons, and ensure evaluations drive trustworthy, measurable gains in performance for production systems.

Qualifications

• Hands-on experience building LLM benchmarks or evals that inform real decisions.
• Strong Python and software engineering; eval code is reproducible and fast.
• Statistical judgment: variance, confidence intervals, sample size, inter-rater agreement.
• Working knowledge of post-training (SFT, RL with verifiable rewards) and how evals feed into it.
• Comfort with SQL and large datasets; you'll work directly from production traces.
• Taste for what is worth measuring.

Responsabilités

• Build benchmarks from real workflows: turn agent traces, customer edge cases and expert labels into versioned benchmark suites for long-horizon agents that run for hours across tools and systems.
• Own the benchmarking effort: decide what we measure, run benchmark suites across models and agent versions, and report results the team makes decisions on.
• Design graders: rubrics, programmatic checks and llm-as-judge, calibrated against human agreement, with a clear view of where each one breaks.
• Close the loop with post-training: turn evals into reward functions and RL environments, and catch reward hacking and contamination before they ship.
• Publish external benchmarks and write-ups on how models perform at real finance work.

Connaissances

LLM benchmarks

Python

Statistical judgment

Post-training evals

SQL

Experiment design

Description du poste

Ramp Labs is seeking a Benchmark Engineer to turn real finance workflows into robust benchmarks and RL environments. You will design graders, convert eval outcomes into reward signals, and publish findings that guide model improvements used across Ramp’s finance tooling.

In this role, you will own the benchmarking lifecycle, run cross-model comparisons, and ensure evaluations drive trustworthy, measurable gains in performance for production systems.