Remote LLM Test Scenario Designer

Mindrift (Toloka); United States; Remote; USD 25,000 - 41,000; Part time

Benefits offered by this job

Remote freelance role

Part-time schedule

Job summary

Mindrift is seeking a freelance remote designer to craft realistic evaluation scenarios for LLM-based agents. You will define gold-standard behaviors, annotate steps, and ensure tests are clear, scored well, and reusable.

You should have a strong analytical mindset, attention to detail, and a current or recent degree in CS/AI/ML/NLP. English proficiency at advanced level (C1+) is required.

Qualifications

• Pursuing or completed degree in CS/SE/DS/AI/NLP or related field.
• Strong analytical mindset and attention to detail.
• Ability to work remotely and learn new methods.

Responsibilities

• Design structured evaluation scenarios based on real-world tasks.
• Define the golden path and acceptable agent behavior.
• Annotate task steps, outputs, and edge cases.
• Collaborate with developers to test scenarios and improve clarity.
• Review agent outputs and adapt tests accordingly.

Skills

Advanced English (C1+)

Education

Bachelor's or Master's degree in CS/SE/DS/AI/ML/NLP or related field

Job description

Mindrift is seeking a freelance remote designer to craft realistic evaluation scenarios for LLM-based agents. You will define gold-standard behaviors, annotate steps, and ensure tests are clear, scored well, and reusable.

You should have a strong analytical mindset, attention to detail, and a current or recent degree in CS/AI/ML/NLP. English proficiency at advanced level (C1+) is required.