LLM Benchmarking Engineer – Finance AI

Ramp Corp.; New York (NY); Sur place; USD 180 000 - 240 000; Plein temps

Avantages offerts par ce poste

Flexible PTO

Centralized home-office equipment

Health and wellness stipend

Budget for intra-office travel

Weekly coffee stipend

Résumé du poste

Ramp Corp. is building the next generation benchmarking platform for finance-focused AI models. As part of Ramp Labs, you will turn real finance workflows into benchmarks and RL environments, design graders, and ensure the evaluation loop improves models in production.

You will collaborate with data engineers, researchers, and product teams to publish benchmarks and practical insights, advancing model performance on expense policy review, invoice coding, and close processes.

Qualifications

• Hands-on experience building LLM benchmarks or evals that people used to make real decisions.
• Strong Python and software engineering; your eval code is reproducible and fast.
• Statistical judgment: variance, confidence intervals, sample size, inter-rater agreement, and knowing when a delta is noise.
• Working knowledge of post-training (SFT, RL with verifiable rewards) and how evals feed into it.
• Comfort with SQL and large datasets; you'll work directly from production traces.
• Taste for what is worth measuring.

Responsabilités

• Build benchmarks from real workflows: turn agent traces, customer edge cases and expert labels into versioned benchmark suites for long-horizon agents that run for hours across tools and systems, plus tool use, multi-step reasoning and financial judgment.
• Own the benchmarking effort: decide what we measure, run benchmark suites across models and agent versions, and report results the team makes decisions on
• Design graders: rubrics, programmatic checks and llm-as-judge, calibrated against human agreement, with a clear view of where each one breaks
• Close the loop with post-training: turn evals into reward functions and RL environments, and catch reward hacking and contamination before they ship
• Publish external benchmarks and write-ups on how models perform at real finance work

Connaissances

Python

SQL

Software engineering

Statistics

Post-training RL

Data analysis

Outils

llm-as-judge

Description du poste

Ramp Corp. is building the next generation benchmarking platform for finance-focused AI models. As part of Ramp Labs, you will turn real finance workflows into benchmarks and RL environments, design graders, and ensure the evaluation loop improves models in production.

You will collaborate with data engineers, researchers, and product teams to publish benchmarks and practical insights, advancing model performance on expense policy review, invoice coding, and close processes.