AI Benchmarking Engineer — Finance & RL Systems

Candidate; New York (NY); On-site; USD 150,000 - 230,000; Full time

Benefits offered by this job

Flexible PTO

Centralized home-office equipment

Health and wellness stipend

Budget for intra-office travel

Weekly coffee stipend

Job summary

Ramp Labs is seeking a highly autonomous researcher/engineer to own benchmarking for AI models that automate finance workflows. You will transform real finance tasks (expense policy review, invoice coding, month-end close) into robust benchmarks and RL environments, build graders, and help the team interpret improvements.

You will publish external benchmarks and ensure evaluation methods remain rigorous as models iterate in production.

Qualifications

• Experience building LLM benchmarks or evals used to inform decisions.
• Strong Python and software engineering; eval code is reproducible and fast.
• Knowledge of statistics: variance, confidence intervals, sample size, inter-rater agreement.

Responsibilities

• Build benchmarks from real workflows and turn them into benchmark suites.
• Own benchmarking: decide metrics, run suites, report results to drive decisions.
• Design graders and llm-as-judge with human-aligned rubrics.
• Convert evals into reward functions and RL environments.
• Publish benchmarks and write-ups on model performance in finance workflows.

Skills

LLM benchmarks

Python

Statistical judgment

Post-training RL

SQL

Benchmarks

Evaluation frameworks

Tools

Jupyter

Job description

Ramp Labs is seeking a highly autonomous researcher/engineer to own benchmarking for AI models that automate finance workflows. You will transform real finance tasks (expense policy review, invoice coding, month-end close) into robust benchmarks and RL environments, build graders, and help the team interpret improvements.

You will publish external benchmarks and ensure evaluation methods remain rigorous as models iterate in production.