Finance AI Benchmarking Engineer (RL & Evals)

Ramp; United States; A distancia; USD 180.000 - 240.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Flexible PTO

Centralized home-office equipment

Health and wellness stipend

Travel budget for intra-office visits

Weekly coffee stipend

Descripción de la vacante

Ramp Labs is seeking a role focused on benchmarking large language models against real finance workflows. You will turn expense policy reviews, invoice coding, and month-end close into benchmarks and RL environments, building graders and evaluative tools to ensure reliable improvements.

You will publish findings and drive end-to-end evaluation loops. Ideal candidates have hands-on experience with LLM benchmarks, strong Python, SQL skills, and a solid grasp of post-training methods.

Formación

• Experience building LLM benchmarks or evals used for real decisions.
• Strong Python and software engineering; eval code is reproducible and fast.
• Statistical judgment: variance, confidence intervals, sample size, inter-rater agreement.
• Working knowledge of post-training (SFT, RL with rewards) and how evals feed into it.
• Comfort with SQL and large datasets; work from production traces.
• Taste for what is worth measuring.

Responsabilidades

• Build benchmarks from real workflows and agent traces for long-horizon tasks.
• Own benchmarking: decide what to measure and report results for team decisions.
• Design graders: rubrics, checks and llm-as-judge with human agreement calibration.
• Close the loop with post-training: turn evals into reward functions and RL environments.
• Publish external benchmarks and write-ups on model performance in finance work.

Conocimientos

Python

Software engineering

LLM benchmarks

Post-training RL

SQL

Data analysis

Educación

Bachelor's in CS/Math/Engineering

Herramientas

Git

Descripción del empleo

Ramp Labs is seeking a role focused on benchmarking large language models against real finance workflows. You will turn expense policy reviews, invoice coding, and month-end close into benchmarks and RL environments, building graders and evaluative tools to ensure reliable improvements.

You will publish findings and drive end-to-end evaluation loops. Ideal candidates have hands-on experience with LLM benchmarks, strong Python, SQL skills, and a solid grasp of post-training methods.