AI Evaluation Engineer (Fully Remote)

Jobgether; United States; Remote; USD 120,000 - 180,000; Full time

Benefits offered by this job

Fully remote

Flexible schedule

Paid training and professional devlop.

Performance-based annual pay increases

negotiable hourly rate

Job summary

Jobgether is hiring to design and build evaluation infrastructure for production-grade AI agents in a fully remote setup. You will develop automated testing systems for complex reasoning and multi-turn conversations, and own the evaluation architecture from ground up.

The role emphasizes Python/TypeScript proficiency, experience with LLM evaluation pipelines, and seamless collaboration across time zones, especially US Central Time. Competitive, remote-friendly compensation offered.

Qualifications

• Experience building LLM evaluation pipelines and integrating tests into CI/CD.
• Excellent English and ability to overlap with US Central Time business hours.
• Fully remote work with flexible schedule.

Responsibilities

• Build CI/CD testing pipelines to validate AI agent behavior.
• Create statistical evaluation frameworks for non-deterministic AI behavior.
• Own the evaluation architecture from ground up in a fully remote environment.

Skills

English

CI/CD testing

LLM evaluation pipelines

English language proficiency

Remote collaboration

Tools

Python

TypeScript

Job description

• Design and build evaluation infrastructure for production-grade AI agents.
• Develop automated testing systems for complex reasoning and multi-turn conversations.
• Own the evaluation architecture from ground up in a fully remote environment.

Key Responsibilities:

• Build CI/CD testing pipelines to validate AI agent behavior.
• Create statistical evaluation frameworks for non-deterministic AI behavior.

Requirements:

• Strong proficiency in Python or TypeScript.
• Experience building LLM evaluation pipelines and integrating tests into CI/CD.
• Excellent English and ability to overlap with US Central Time business hours.
• Fully remote work with flexible schedule.
• Paid training and professional development opportunities.
• Performance-based annual pay increases and negotiable hourly rate.

Jobgether

A company building production-grade AI agents and systems. They operate in a fully remote, globally distributed environment with a culture focused on innovation, autonomy, and continuous improvement.