LLM Evaluation Scenario Architect

Mindrift; Mississippi; Remote; USD 90,921 - 129,494; Part time

Benefits offered by this job

Competitive hourly rates

Flexible work hours

Valuable experience for portfolio

Job summary

A leading AI solutions company seeks a candidate for evaluating LLM-based agents. The role involves designing scenarios, creating test cases, and analyzing decision-making processes. A Bachelor's or Master's in relevant fields is required, alongside skills in data analysis and communication. The position offers flexible, remote work with rates up to $80/hour, allowing you to influence AI model developments while fitting around your current commitments.

Qualifications

• Bachelor's and/or Master's in Computer Science, Software Engineering, AI or related fields.
• Background in QA, software testing, or data analysis.
• Good understanding of test design principles like reproducibility and edge cases.

Responsibilities

• Design realistic evaluation scenarios for LLM-based agents.
• Create structured test cases simulating human workflows.
• Analyze agent logs and decision paths.

Skills

Analytical mindset

Attention to detail

Strong written communication skills in English

Understanding of test design principles

Curiosity about AI-generated content

Education

Bachelor's and/or Master's Degree in relevant fields

Tools

Python

JavaScript

JSON/YAML

Job description

A leading AI solutions company seeks a candidate for evaluating LLM-based agents. The role involves designing scenarios, creating test cases, and analyzing decision-making processes. A Bachelor's or Master's in relevant fields is required, alongside skills in data analysis and communication. The position offers flexible, remote work with rates up to $80/hour, allowing you to influence AI model developments while fitting around your current commitments.