Remote AI Evaluator Research Engineer (Equity)

Ragas; United States; Remote; USD 60,000 - 200,000; Full time

Benefits offered by this job

Remote role

Job summary

Ragas seeks researchers to develop methods that improve evaluation of applications built with large language models. You will work on optimization, synthetic test data, and adapting models to serve as evaluators, aiming to align automated assessments with human judgments.

Responsibilities include designing optimization algorithms, creating synthetic test cases, fine-tuning evaluators, and applying research findings to practical evaluation problems.

Qualifications

• Extensive experience with large language models.
• Experience with synthetic data generation and optimization.
• Interest in reading and critically assessing research with openness to new approaches.
• Evidence of open-source contributions or Kaggle participation.

Responsibilities

• Design optimization algorithms to improve agreement between model-based evaluators and human assessments.
• Create methods for generating synthetic test cases for language model applications, including retrieval-augmented and tool-using workflows.
• Fine-tune and align language models for use as custom evaluators.
• Apply findings from research to practical evaluation problems and develop new solutions.

Skills

Large language models

Synthetic data generation

Optimization

Open-source contributions

Job description

Ragas seeks researchers to develop methods that improve evaluation of applications built with large language models. You will work on optimization, synthetic test data, and adapting models to serve as evaluators, aiming to align automated assessments with human judgments.

Responsibilities include designing optimization algorithms, creating synthetic test cases, fine-tuning evaluators, and applying research findings to practical evaluation problems.