AI Evaluation Engineer

Jobgether; Saudi Arabia; Teletrabalho; SAR 165 000 - 331 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Fully remote work

Flexible schedule

Professional development

Modern technology

Project opportunities

Performance-based raises

Negotiable compensation

Independent contractor

Flexible working environment

Resumo da oferta

Jobgether is seeking a founding AI Evaluation Engineer based in Saudi Arabia to design and build scalable evaluation pipelines for AI agents. You will ensure reliability, safety, and efficiency across automated testing systems, including multi-turn conversations, API interactions, and code execution, in a remote, globally distributed environment.

The role involves building observability, regression testing, and adversarial scenarios to identify weaknesses before deployment, with ownership of the

Qualificações

• Strong proficiency in Python or TypeScript for API testing, headless execution, and asynchronous programming.
• Experience building LLM evaluation pipelines, AI testing frameworks, or statistical testing systems.
• Experience integrating automated tests into CI/CD environments, especially GitHub Actions.
• Strong expertise validating structured data: JSON schema, OpenAPI specs, function-calling payloads.
• Familiarity with observability tools such as OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave.
• Ability to design and own an evaluation architecture from scratch.

Responsabilidades

• Design, build, and maintain automated CI/CD testing pipelines for AI agents.
• Develop golden datasets with regression prompts and edge cases.
• Implement fast evaluation checks including JSON schema validation and API status.
• Create test suites for multi-turn agent conversations and context retention.
• Design adversarial testing scenarios for prompt injection and unexpected tool responses.
• Build statistical evaluation frameworks for non-deterministic AI behavior.
• Integrate evaluation workflows into GitHub Actions or similar CI systems.
• Implement observability and tracing solutions to monitor agent execution.
• Optimize evaluation efficiency with caching and mocking to reduce LLM calls.
• Validate outputs including JSON schemas and function-calling payloads.
• Evaluate performance under load and varying endpoint behavior.
• Independently design and own the evaluation architecture.

Conhecimentos

Python/TypeScript

API testing

CI/CD

GitHub Actions

OpenTelemetry

LangSmith

Arize Phoenix

Weave (W&B)

LangChain

LlamaIndex

CrewAI

Vector embeddings

Load testing

JSON schema

Ferramentas

LangSmith

Arize Phoenix

W&B Weave

LangChain

LlamaIndex

AutoGen

CrewAI

Descrição da oferta de emprego

This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for an AI Evaluation Engineer based in Saudi Arabia.

As a founding AI Evaluation Engineer, you will design and build the evaluation, testing, and observability infrastructure that ensures production-grade AI agents perform reliably, safely, and efficiently. You will develop automated testing systems for complex reasoning, multi-turn conversations, and agent interactions involving APIs, databases, and code execution. Working across AI engineering, software testing, and infrastructure, you will create scalable evaluation pipelines that go beyond traditional QA and manual prompt testing. You will establish robust regression testing, statistical evaluation methods, and adversarial scenarios to identify weaknesses before deployment. This is an opportunity to take ownership of an evaluation architecture from the ground up while working with modern AI technologies in a fully remote, globally distributed environment. Your work will directly contribute to the reliability, security, and continuous improvement of advanced AI systems.

Accountabilities

• Design, build, and maintain automated CI/CD testing pipelines to validate AI agent behavior before deployment and throughout the development lifecycle.

• Develop and maintain golden datasets containing regression prompts, edge cases, and synthetic scenarios to measure agent accuracy, safety, performance, and latency.

• Implement fast, cost-effective, deterministic evaluation checks before introducing LLM-as-a-judge approaches, including JSON schema validation, regular expressions, API status checks, and embedding similarity.

• Create comprehensive test suites for multi-turn agent conversations, evaluating context retention, state consistency, reasoning behavior, and safeguards against infinite tool-calling loops.

• Design and implement adversarial testing scenarios to assess resilience against prompt injection, malformed inputs, unexpected tool responses, and other potential security vulnerabilities.

• Build statistical evaluation frameworks to measure non-deterministic AI behavior using pass rates across multiple runs, confidence intervals, and other quantitative reliability metrics.

• Integrate evaluation workflows into GitHub Actions or comparable CI systems using SDK-level execution and automated testing practices.

• Implement observability and tracing solutions using tools such as OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave, or equivalent platforms to monitor agent execution and diagnose failures.

• Optimize evaluation efficiency and cost through caching, mocking, and other techniques that reduce unnecessary LLM API calls.

• Validate structured outputs, including JSON schemas, OpenAPI specifications, and function-calling payloads, to ensure agents interact reliably with external systems.

• Evaluate and improve agent performance under load, accounting for variable response times and non-deterministic endpoint behavior.

• Independently design and own the evaluation architecture, establishing best practices, testing standards, and scalable processes for ongoing quality assurance.

Requirements

• Strong proficiency in Python or TypeScript, particularly for API testing, headless execution, asynchronous programming, and automated testing systems.

• Hands-on experience building LLM evaluation pipelines, AI testing frameworks, or statistical testing systems capable of measuring performance across repeated runs.

• Experience integrating automated tests into CI/CD environments, particularly GitHub Actions or similar platforms, with SDK-level execution.

• Strong expertise in validating structured data and outputs, including JSON schema, OpenAPI specifications, and function-calling payloads.

• Familiarity with observability and tracing tools such as OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave, or custom logging solutions.

• Ability to independently design, implement, and maintain an evaluation architecture from scratch, with a strong sense of ownership and technical judgment.

• Excellent written and spoken English, with the ability to communicate technical findings clearly in a distributed team.

• Availability to overlap with US Central Time business hours for at least three hours each day.

• Experience with code execution sandboxes or AI agent frameworks such as LangChain, LlamaIndex, AutoGen, or CrewAI.

• Strong understanding of embedding models, vector databases, and their applications in AI evaluation and similarity-based testing.

• Experience with load testing non-deterministic endpoints and assessing system behavior under varying workloads.

• A proactive, analytical approach to problem-solving, with the ability to work independently, adapt to evolving technologies, and continuously improve testing methodologies.

Benefits

• Fully remote work: Work from anywhere with a reliable internet connection, with the flexibility to choose your preferred time zone and working hours.

• Flexible schedule: Organize your working day around your productivity while maintaining the required overlap with US Central Time.

• Professional development: Access paid training and other opportunities to expand your technical expertise and support your career growth.

• Modern technology: Work with contemporary AI tools, frameworks, and engineering practices in an environment that values innovation and continuous improvement.

• Consistent project opportunities: Benefit from a steady flow of engaging and challenging product development work.

• Performance recognition: Strong performance and dedication may be rewarded through annual pay increases.

• Negotiable compensation: Your starting hourly rate will be determined based on your skills, experience, and qualifications.

• Independent contractor arrangement: Compensation is based on hours worked, with payments made monthly via wire transfer.

• Flexible working environment: Enjoy the autonomy to work in a setting that supports your focus, independence, and professional development.

Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.