Applied Research Engineer

Ragas; United States; Remote; USD 60,000 - 200,000; Full time

Benefits offered by this job

Remote role

Job summary

Ragas seeks researchers to develop methods that improve evaluation of applications built with large language models. You will work on optimization, synthetic test data, and adapting models to serve as evaluators, aiming to align automated assessments with human judgments.

Responsibilities include designing optimization algorithms, creating synthetic test cases, fine-tuning evaluators, and applying research findings to practical evaluation problems.

Qualifications

• Extensive experience with large language models.
• Experience with synthetic data generation and optimization.
• Interest in reading and critically assessing research with openness to new approaches.
• Evidence of open-source contributions or Kaggle participation.

Responsibilities

• Design optimization algorithms to improve agreement between model-based evaluators and human assessments.
• Create methods for generating synthetic test cases for language model applications, including retrieval-augmented and tool-using workflows.
• Fine-tune and align language models for use as custom evaluators.
• Apply findings from research to practical evaluation problems and develop new solutions.

Skills

Large language models

Synthetic data generation

Optimization

Open-source contributions

Job description

Role overview

Develop research-informed methods that make it easier to evaluate applications built with large language models. The work spans optimization, synthetic test data, and adapting models to serve as evaluators, with a focus on making automated assessments align more closely with human judgments.

Responsibilities

• Design optimization algorithms to improve agreement between model-based evaluators and human assessments.
• Create methods for generating synthetic test cases for language model applications, including retrieval-augmented and tool-using workflows.
• Fine-tune and align language models for use as custom evaluators.
• Apply findings from research to practical evaluation problems and develop new solutions.

Requirements

• Extensive experience working with large language models.
• Experience with synthetic data generation and optimization.
• Interest in reading and critically assessing research while remaining open to new approaches.
• Evidence of interest in open source or competitive data science, such as contributions to open-source projects or Kaggle participation.

Benefits and work setup

• Remote role.
• Advertised compensation range: $60,000–$200,000, plus 0.5%–2.5% equity.