AI Benchmarking Engineer for RL & Finance Evals

Wwshemi; New York, San Francisco (NY, CA); Hybride; USD 150 000 - 210 000; Plein temps

Avantages offerts par ce poste

Flexible PTO

Centralized home-office equipment

Health and wellness stipend

Budget for intra-office travel

Weekly coffee stipend

Résumé du poste

Ramp Labs seeks a benchmarking specialist to turn real finance workflows into scalable benchmarks for long-horizon agents. You will build graders and evaluators, connecting post-training with reward functions and RL environments.

You’ll own the benchmarking effort, decide what metrics to measure, and report results to drive team decisions. Strong Python, SQL, and statistical judgment are essential.

Qualifications

• Hands-on experience building LLM benchmarks or evals used to inform real decisions.
• Strong Python and software engineering; eval code is reproducible and fast.
• Statistical judgment: variance, confidence intervals, sample size, inter-rater agreement.

Responsabilités

• Build benchmarks from real workflows and expert labels for long-horizon agents.
• Own benchmarking—decide what to measure and report results for decision-making.
• Design graders: rubrics, programmatic checks and llm-as-judge with human agreement.
• Close the loop with post-training: turn evals into reward functions and RL environments.
• Publish external benchmarks and write-ups on model performance in finance work.

Connaissances

LLM benchmarks

Python

Software engineering

Statistical judgment

SQL

Formation

Bachelor’s degree or higher

Outils

LLM Eval Harness

Helm

Description du poste

Ramp Labs seeks a benchmarking specialist to turn real finance workflows into scalable benchmarks for long-horizon agents. You will build graders and evaluators, connecting post-training with reward functions and RL environments.

You’ll own the benchmarking effort, decide what metrics to measure, and report results to drive team decisions. Strong Python, SQL, and statistical judgment are essential.