Remote AI Evaluation Architect: Build Scalable Pipelines

Jobgether; Ireland; Remote; EUR 83,000 - 138,000; Full time

Benefits offered by this job

Fully remote work

Flexible schedule

Professional development

Modern technology

Independent contractor arrangement

Job summary

Jobgether is seeking an AI Evaluation Engineer based in Ireland to join as a founding member, designing and building the evaluation, testing, and observability infrastructure for production-grade AI agents.

You will develop automated testing systems for complex reasoning, multi-turn conversations, and tool interactions, establishing robust regression and adversarial testing with full ownership of the evaluation architecture.

Qualifications

• Strong Python or TypeScript programming for API testing and automation.
• Experience building LLM evaluation pipelines and AI testing frameworks.
• Experience integrating tests into CI/CD, especially GitHub Actions.
• Experience validating JSON schemas, OpenAPI specs, and function payloads.
• Familiarity with observability tools and tracing platforms.

Responsibilities

• Design, build, and maintain automated CI/CD testing pipelines for AI agents.
• Develop golden datasets for regression prompts and edge cases.
• Implement evaluation checks including JSON schema validation and API status checks.
• Create test suites for multi-turn agent conversations and tool interactions.
• Design adversarial tests to probe prompt injections and security risks.
• Build statistical evaluation frameworks with multiple runs and confidence intervals.
• Integrate evaluation workflows into CI systems like GitHub Actions.
• Implement observability and tracing to monitor agent execution and diagnose failures.
• Optimize evaluation efficiency to reduce LLM calls.
• Validate structured outputs and payloads for external systems.

Skills

Python

TypeScript

API testing

CI/CD

GitHub Actions

JSON schema

OpenTelemetry

LangChain

LlamaIndex

CrewAI

Tools

GitHub Actions

LangSmith

Arize Phoenix

W&B Weave

OpenTelemetry

Job description

Jobgether is seeking an AI Evaluation Engineer based in Ireland to join as a founding member, designing and building the evaluation, testing, and observability infrastructure for production-grade AI agents.

You will develop automated testing systems for complex reasoning, multi-turn conversations, and tool interactions, establishing robust regression and adversarial testing with full ownership of the evaluation architecture.