Remote AI Agent Evaluation Engineer (Freelance)

aitrainer; United States; Remote; USD 27,552,000 - 41,328,000; Part time

Job summary

Mindrift seeks experienced software developers to design AI evaluation tasks. You will create realistic development environments, craft intermediate-state tasks, and write tests to validate agent solutions. This is a part-time, remote freelance role focused on robust evaluation criteria and fair task design.

Candidates should have 5+ years in software development with strong Python/React stacks and experience writing tests. English proficiency (B2+) is required.

Qualifications

• 5+ years of professional experience in QA-automation/testing or cybersecurity roles.
• Master’s Degree in CS/SE/AI/NLP/IS or related fields; Bachelor’s allowed with 5 years experience.
• English proficiency required, minimum B2+.

Responsibilities

• Create challenging tasks and evaluation criteria within realistic simulated environments.
• Design prompts, define what 'solved' means, and ensure solvability by an AI agent.
• Write tests that verify agent solutions and accept all valid approaches.
• Iterate on tasks and tests based on QA feedback and refine until evaluation is fair.

Skills

5+ years in software development

Python (FastAPI)

JavaScript/TypeScript (React)

Docker

Postgres

Kafka

Redis

Testing (functional)

English proficiency - B2+

Education

Master’s Degree in CS/SE/AI/ML/NLP/IS or related fields

Bachelor’s degree with 5 years experience accepted

Job description

Mindrift seeks experienced software developers to design AI evaluation tasks. You will create realistic development environments, craft intermediate-state tasks, and write tests to validate agent solutions. This is a part-time, remote freelance role focused on robust evaluation criteria and fair task design.

Candidates should have 5+ years in software development with strong Python/React stacks and experience writing tests. English proficiency (B2+) is required.