AI Evaluation Engineer

Jobgether; Brasil; Teletrabalho; BRL 90 000 - 150 000; Tempo integral

Resumo da oferta

Jobgether is seeking an AI Evaluation Engineer based in Brazil to join a partner company’s pioneering AI program. As a founding engineer, you will design evaluation, testing, and observability infrastructure to ensure production-grade AI agents perform reliably, safely, and efficiently in a fully remote, globally distributed environment.

You will create scalable evaluation pipelines, automate testing for complex reasoning and multi-turn conversations, and implement robust regression testing,

Responsabilidades

• Design and implement evaluation infrastructure for AI agents.
• Build automated testing systems for complex reasoning and multi-turn conversations.
• Develop observability pipelines to monitor performance across APIs, databases, and code execution.
• Define regression tests, statistical evaluation methods, and adversarial scenarios.

Descrição da oferta de emprego

Description:

This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for an AI Evaluation Engineer based in Brazil.

As a founding AI Evaluation Engineer, you will design and build the evaluation, testing, and observability infrastructure that ensures production‑grade AI agents perform reliably, safely, and efficiently. You will develop automated testing systems for complex reasoning, multi-turn conversations, and agent interactions involving APIs, databases, and code execution. Working across AI engineering, software testing, and infrastructure, you will create scalable evaluation pipelines that go beyond traditional QA and manual prompt testing. You will establish robust regression testing, statistical evaluation methods, and adversarial scenarios to identify weaknesses before deployment. This is an opportunity to take ownership of an evaluation architecture from the ground up while working with modern AI technologies in a fully remote, globally distributed environment. Your work will directly contribute to the reliability, security, and continuous improvement of advanced AI systems.