AI Benchmarking Engineer for Finance Tech

Warmada; New York (NY); On-site; USD 150,000 - 190,000; Full time

Benefits offered by this job

Flexible PTO

Home-office equipment ordering

Health and wellness stipend

Intra-office travel budget

Weekly coffee stipend

Job summary

Ramp Labs is seeking a dedicated benchmarking lead to turn real finance workflows into versioned benchmarks and RL environments. You will design graders and reward structures, ensuring evals drive real improvements across long-horizon agents in complex financial tasks.

You’ll own end-to-end benchmarking from data collection to reporting results used by the team to make decisions. This role blends research with production-grade tooling.

Qualifications

• Hands-on experience building LLM benchmarks or evals that people used to make real decisions.
• Strong Python and software engineering; your eval code is reproducible and fast.
• Statistical judgment: variance, confidence intervals, sample size, inter-rater agreement, and knowing when a delta is noise.
• Working knowledge of post-training (SFT, RL with verifiable rewards) and how evals feed into it.
• Comfort with SQL and large datasets; you’ll work directly from production traces.
• Taste for what is worth measuring.

Responsibilities

• Build benchmarks from real workflows: agent traces, edge cases and labels into versioned benchmark suites.
• Own the benchmarking effort: decide what we measure, run benchmarks across models and versions.
• Design graders: rubrics, programmatic checks and llm-as-judge, calibrated against human agreement.
• Close the loop with post-training: turn evals into reward functions and RL environments.
• Publish external benchmarks and write-ups on model performance at real finance work.

Skills

LLM benchmarks

Python

Software engineering

Statistics

Post-training

SQL

Job description

Ramp Labs is seeking a dedicated benchmarking lead to turn real finance workflows into versioned benchmarks and RL environments. You will design graders and reward structures, ensuring evals drive real improvements across long-horizon agents in complex financial tasks.

You’ll own end-to-end benchmarking from data collection to reporting results used by the team to make decisions. This role blends research with production-grade tooling.