Founding AI Evaluation Engineer — Remote Testing Architect

Jobgether; Saudi Arabia; Remote; SAR 165,000 - 331,000; Full time

Benefits offered by this job

Fully remote work

Flexible schedule

Professional development

Modern technology

Project opportunities

Performance-based raises

Negotiable compensation

Independent contractor

Flexible working environment

Job summary

Jobgether is seeking a founding AI Evaluation Engineer based in Saudi Arabia to design and build scalable evaluation pipelines for AI agents. You will ensure reliability, safety, and efficiency across automated testing systems, including multi-turn conversations, API interactions, and code execution, in a remote, globally distributed environment.

The role involves building observability, regression testing, and adversarial scenarios to identify weaknesses before deployment, with ownership of the

Qualifications

• Strong proficiency in Python or TypeScript for API testing, headless execution, and asynchronous programming.
• Experience building LLM evaluation pipelines, AI testing frameworks, or statistical testing systems.
• Experience integrating automated tests into CI/CD environments, especially GitHub Actions.
• Strong expertise validating structured data: JSON schema, OpenAPI specs, function-calling payloads.
• Familiarity with observability tools such as OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave.
• Ability to design and own an evaluation architecture from scratch.

Responsibilities

• Design, build, and maintain automated CI/CD testing pipelines for AI agents.
• Develop golden datasets with regression prompts and edge cases.
• Implement fast evaluation checks including JSON schema validation and API status.
• Create test suites for multi-turn agent conversations and context retention.
• Design adversarial testing scenarios for prompt injection and unexpected tool responses.
• Build statistical evaluation frameworks for non-deterministic AI behavior.
• Integrate evaluation workflows into GitHub Actions or similar CI systems.
• Implement observability and tracing solutions to monitor agent execution.
• Optimize evaluation efficiency with caching and mocking to reduce LLM calls.
• Validate outputs including JSON schemas and function-calling payloads.
• Evaluate performance under load and varying endpoint behavior.
• Independently design and own the evaluation architecture.

Skills

Python/TypeScript

API testing

CI/CD

GitHub Actions

OpenTelemetry

LangSmith

Arize Phoenix

Weave (W&B)

LangChain

LlamaIndex

CrewAI

Vector embeddings

Load testing

JSON schema

Tools

LangSmith

Arize Phoenix

W&B Weave

LangChain

LlamaIndex

AutoGen

CrewAI

Job description

Jobgether is seeking a founding AI Evaluation Engineer based in Saudi Arabia to design and build scalable evaluation pipelines for AI agents. You will ensure reliability, safety, and efficiency across automated testing systems, including multi-turn conversations, API interactions, and code execution, in a remote, globally distributed environment.

The role involves building observability, regression testing, and adversarial scenarios to identify weaknesses before deployment, with ownership of the