Remote AI Evaluation Engineer - Build Scalable Testing

Jobgether; Portugal; Remote; EUR 83,000 - 124,000; Full time

Benefits offered by this job

Fully remote

Flexible schedule

Professional development

Modern technology

Independent contractor

Flexible working environment

Job summary

Jobgether is seeking an AI Evaluation Engineer to design evaluation, testing, and observability infrastructure for production-grade AI agents. You will create scalable evaluation pipelines, automate tests, and implement robust regression and adversarial scenarios, all in a fully remote, globally distributed setting.

You will own the evaluation architecture, build fast checks, validate JSON schemas and function-calling payloads, and collaborate across AI engineering, software testing, and

Qualifications

• Proficiency in Python or TypeScript for API testing and automation.

Responsibilities

• Design, build automated CI/CD testing pipelines for AI agents.

Skills

Python

TypeScript

API testing

CI/CD

Observability

JSON schema

LLM evaluation

Data analysis

LangSmith

GitHub Actions

LangChain

LlamaIndex

CrewAI

OpenTelemetry

Tools

GitHub Actions

LangSmith

Arize Phoenix

W&B Weave

OpenTelemetry

LangChain

LlamaIndex

Job description

Jobgether is seeking an AI Evaluation Engineer to design evaluation, testing, and observability infrastructure for production-grade AI agents. You will create scalable evaluation pipelines, automate tests, and implement robust regression and adversarial scenarios, all in a fully remote, globally distributed setting.

You will own the evaluation architecture, build fast checks, validate JSON schemas and function-calling payloads, and collaborate across AI engineering, software testing, and