Remote AI Evaluation Architect — Build Reliable Agents

Jobgether; South Africa; Remote; ZAR 248,000 - 386,000; Full time

Benefits offered by this job

Fully remote

Flexible schedule

Professional development

Modern technology

Independent contractor arrangement

Monthly payments by wire transfer

Continuous project opportunities

Performance-based raises

negotiable compensation

Job summary

Jobgether, on behalf of a partner company, seeks an AI Evaluation Engineer based in South Africa. You will design automated evaluation, testing, and observability infrastructure for production-grade AI agents, focusing on reliability, safety, and efficiency.

You will build pipelines for multi-turn conversations, integrate with CI/CD, implement adversarial testing, and define robust statistical evaluation methods.

Qualifications

• Strong proficiency in Python or TypeScript, particularly for API testing, headless execution, asynchronous programming, and automated testing systems.
• Hands-on experience building LLM evaluation pipelines, AI testing frameworks, or statistical testing systems capable of measuring performance across repeated runs.
• Experience integrating automated tests into CI/CD environments, particularly GitHub Actions or similar platforms, with SDK-level execution.
• Strong expertise in validating structured data and outputs, including JSON schema, OpenAPI specifications, and function-calling payloads.
• Familiarity with observability and tracing tools such as OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave, or custom logging solutions.
• Ability to independently design, implement, and maintain an evaluation architecture from scratch, with a strong sense of ownership and technical judgment.
• Excellent written and spoken English, with the ability to communicate technical findings clearly in a distributed team.
• Availability to overlap with US Central Time business hours for at least three hours each day.
• Experience with code execution sandboxes or AI agent frameworks such as LangChain, LlamaIndex, AutoGen, or CrewAI.
• Strong understanding of embedding models, vector databases, and their applications in AI evaluation and similarity-based testing.
• Experience with load testing non-deterministic endpoints and assessing system behavior under varying workloads.
• A proactive, analytical approach to problem-solving, with the ability to work independently, adapt to evolving technologies, and continuously improve testing methodologies.

Responsibilities

• Design, build, and maintain automated CI/CD testing pipelines to validate AI agent behavior before deployment and throughout the development lifecycle.
• Develop and maintain golden datasets containing regression prompts, edge cases, and synthetic scenarios to measure agent accuracy, safety, performance, and latency.
• Implement fast, cost-effective, deterministic evaluation checks before introducing LLM-as-a-judge approaches, including JSON schema validation, regular expressions, API status checks, and embedding similarity.
• Create comprehensive test suites for multi-turn agent conversations, evaluating context retention, state consistency, reasoning behavior, and safeguards against infinite tool-calling loops.
• Design and implement adversarial testing scenarios to assess resilience against prompt injection, malformed inputs, unexpected tool responses, and other potential security vulnerabilities.
• Build statistical evaluation frameworks to measure non-deterministic AI behavior using pass rates across multiple runs, confidence intervals, and other quantitative reliability metrics.
• Integrate evaluation workflows into GitHub Actions or comparable CI systems using SDK-level execution and automated testing practices.
• Implement observability and tracing solutions using tools such as OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave, or equivalent platforms to monitor agent execution and diagnose failures.
• Optimize evaluation efficiency and cost through caching, mocking, and other techniques that reduce unnecessary LLM API calls.
• Validate structured outputs, including JSON schemas, OpenAPI specifications, and function-calling payloads, to ensure agents interact reliably with external systems.
• Evaluate and improve agent performance under load, accounting for variable response times and non-deterministic endpoint behavior.
• Independently design and own the evaluation architecture, establishing best practices, testing standards, and scalable processes for ongoing quality assurance.

Skills

Python/TypeScript

API testing

CI/CD pipelines

JSON schema validation

Observability/Tracing

OpenTelemetry

LangChain/LlamaIndex

Embedding models

Vector databases

English proficiency

Independent ownership

Tools

OpenTelemetry

LangSmith

Arize Phoenix

W&B Weave

Job description

Jobgether, on behalf of a partner company, seeks an AI Evaluation Engineer based in South Africa. You will design automated evaluation, testing, and observability infrastructure for production-grade AI agents, focusing on reliability, safety, and efficiency.

You will build pipelines for multi-turn conversations, integrate with CI/CD, implement adversarial testing, and define robust statistical evaluation methods.