Remote AI Evaluation Engineer - Build AI Testing Pipelines

Jobgether; España; Remote; EUR 69,000 - 124,000; Full time

Benefits offered by this job

Fully remote

Flexible schedule

Professional development

Modern tools & practices

Contractor payments monthly

Job summary

Jobgether is seeking a founding AI Evaluation Engineer based in Spain to design and build scalable evaluation and observability infrastructure for AI agents. You will create automated testing for complex reasoning, multi-turn conversations, and API interactions, driving reliability and safety across production systems.

You will own the evaluation architecture end-to-end, implement robust regression tests, cost-aware optimizations, and adversarial scenarios, while collaborating with distributed

Qualifications

• Proficient in Python or TypeScript for API testing and automated testing systems.
• Experience building LLM evaluation pipelines or AI testing frameworks.
• Experience integrating tests into CI/CD, esp. GitHub Actions.
• Strong validation of JSON schemas, OpenAPI, and function-calling payloads.

Responsibilities

• Design and maintain automated CI/CD testing pipelines for AI agents.
• Develop golden datasets and evaluation metrics for accuracy, safety, latency.
• Create test suites for multi-turn conversations and tool usage loops.
• Implement adversarial testing to assess prompt robustness and security.

Skills

Python

TypeScript

API testing

CI/CD

GitHub Actions

Observability

OpenTelemetry

LangSmith

Arize Phoenix

Tools

LangChain

LlamaIndex

AutoGen

CrewAI

Job description

Jobgether is seeking a founding AI Evaluation Engineer based in Spain to design and build scalable evaluation and observability infrastructure for AI agents. You will create automated testing for complex reasoning, multi-turn conversations, and API interactions, driving reliability and safety across production systems.

You will own the evaluation architecture end-to-end, implement robust regression tests, cost-aware optimizations, and adversarial scenarios, while collaborating with distributed