Founding AI Evaluation Engineer — Remote

Jobgether; United Arab Emirates; Remote; AED 248,000 - 386,000; Full time

Benefits offered by this job

Fully remote work

Flexible schedule

Professional development

Modern AI tooling

Independent contractor arrangement

Competitive compensation

Job summary

Jobgether is seeking a founding AI Evaluation Engineer to build the end-to-end evaluation and observability framework for production-grade AI agents. The role involves designing automated testing, regression datasets, and adversarial scenarios across multi-turn conversations and API interactions, all in a fully remote, globally distributed environment.

You will implement fast deterministic checks, validate JSON schemas and function payloads, and integrate pipelines with CI/CD tools like GitHub

Qualifications

• Proficient in Python/TypeScript for API testing and automation.
• Experience building LLM evaluation pipelines or AI testing frameworks.
• Experience integrating tests into CI/CD (GitHub Actions preferred).
• Strong validation of structured data: JSON schema, OpenAPI, function-calling payloads.
• Familiar with observability/tracing tools (OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave).
• Ability to design end-to-end evaluation architecture from scratch.

Responsibilities

• Design automated CI/CD testing pipelines for AI agent evaluation.
• Create golden datasets for regression prompts and edge cases.
• Implement fast deterministic checks including JSON schema validation and API status checks.
• Build test suites for multi-turn agent conversations and tool use.
• Design adversarial tests for prompt injection and malformed inputs.
• Develop statistical evaluation methods with multiple runs and confidence intervals.
• Integrate evaluation workflows into GitHub Actions or similar systems.
• Establish observability and tracing to monitor agent execution.

Skills

Python

TypeScript

API testing

CI/CD

GitHub Actions

Observability

Tools

OpenTelemetry

LangSmith

Arize Phoenix

Weights & Biases Weave

LangChain

LlamaIndex

AutoGen

CrewAI

Job description

Jobgether is seeking a founding AI Evaluation Engineer to build the end-to-end evaluation and observability framework for production-grade AI agents. The role involves designing automated testing, regression datasets, and adversarial scenarios across multi-turn conversations and API interactions, all in a fully remote, globally distributed environment.

You will implement fast deterministic checks, validate JSON schemas and function payloads, and integrate pipelines with CI/CD tools like GitHub