Remote AI Evaluation Engineer: Build Testing Pipelines

Jobgether; Turkey; Remote; TRY 4,094,000 - 8,188,000; Full time

Benefits offered by this job

Fully remote work

Flexible schedule

Professional development

Modern technology stack

Independent contractor arrangement

Consistent project opportunities

Performance recognition

Negotiable compensation

Flexible working environment

Job summary

Jobgether is seeking a founding AI Evaluation Engineer to design and build scalable evaluation infrastructure for production-grade AI agents. You will create automated testing for complex reasoning, multi-turn conversations, and API interactions, while integrating observability and efficiency improvements.

Remote, globally distributed role requiring overlap with US Central Time daily. You will own the evaluation architecture end-to-end, driving reliability, safety, and continuous improvement

Qualifications

• Strong proficiency in Python or TypeScript for API testing, headless execution, and asynchronous code.
• Hands-on experience building LLM evaluation pipelines or statistical testing systems.
• Experience integrating automated tests into CI/CD, esp. GitHub Actions, with SDK-level execution.
• Strong ability to validate structured data, JSON schemas, OpenAPI specs, and function-calling payloads.
• Familiarity with observability/tracing tools such as OpenTelemetry, LangSmith, Arize Phoenix, Weave, or equivalents.
• Ability to design and maintain evaluation architectures from scratch with ownership and judgment.
• Excellent written and spoken English with clear communication in distributed teams.
• Availability to overlap with US Central Time by at least three hours daily.
• Experience with code execution sandboxes or AI frameworks like LangChain, LlamaIndex, AutoGen, or CrewAI.
• Strong understanding of embedding models, vector databases, and similarity-based testing.
• Experience load-testing non-deterministic endpoints and varying workloads.
• Proactive, analytical problem-solver capable of independent work and evolving technologies.

Responsibilities

• Design, build, and maintain automated CI/CD testing pipelines to validate AI agent behavior before deployment and throughout the lifecycle.
• Develop and maintain golden datasets containing regression prompts, edge cases, and synthetic scenarios for accuracy, safety, and latency.
• Implement fast, deterministic evaluation checks before LLM-as-a-judge approaches, including JSON schema validation and API status checks.
• Create comprehensive test suites for multi-turn agent conversations, evaluating context retention and safeguards against loops.
• Design adversarial testing scenarios to assess resilience to prompt injection and malformed inputs.
• Build statistical evaluation frameworks to measure non-deterministic AI behavior across multiple runs and confidence intervals.
• Integrate evaluation workflows into GitHub Actions or similar CI systems with SDK-level execution.
• Implement observability and tracing using OpenTelemetry, LangSmith, Arize Phoenix, Weave, or equivalents to diagnose failures.
• Optimize evaluation efficiency via caching and mocking to reduce LLM API calls.
• Validate structured outputs and function-calling payloads to ensure reliable external interactions.
• Evaluate agent performance under load and varying response times.
• Independently own the evaluation architecture, establishing best practices and scalable QA processes.

Skills

Python/TypeScript

API testing

CI/CD

LLM testing

OpenTelemetry

LangSmith

GitHub Actions

LangChain

LlamaIndex

CrewAI

Tools

GitHub Actions

LangChain

LlamaIndex

AutoGen

CrewAI

Job description

Jobgether is seeking a founding AI Evaluation Engineer to design and build scalable evaluation infrastructure for production-grade AI agents. You will create automated testing for complex reasoning, multi-turn conversations, and API interactions, while integrating observability and efficiency improvements.

Remote, globally distributed role requiring overlap with US Central Time daily. You will own the evaluation architecture end-to-end, driving reliability, safety, and continuous improvement