Founding Remote AI Evaluation Engineer

Jobgether; México; Remote; MXN 1,526,000 - 3,053,000; Full time

Benefits offered by this job

Fully remote

Flexible schedule

Professional development

Modern technology stack

Independent contractor arrangement

Competitive compensation

Job summary

Jobgether is seeking a founding AI Evaluation Engineer based in Mexico to design and build evaluation, testing, and observability infrastructure for production-grade AI agents. You will create scalable evaluation pipelines, regression testing, and adversarial scenarios to ensure reliability, safety, and efficiency across multi-turn conversations and API interactions.

As an independent contractor working remotely, you will own the evaluation architecture end-to-end, integrating with GitHub

Qualifications

• Proficient in Python or TS for API testing and asynchronous code.
• Experience building LLM evaluation pipelines or AI testing frameworks.
• Experience integrating tests into CI/CD, esp. GitHub Actions.
• Strong validation of JSON schemas, OpenAPI specs, and function-calling payloads.
• Familiarity with observability tools like OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave.

Responsibilities

• Design, build, and maintain automated CI/CD testing pipelines for AI agents.
• Develop golden datasets with regression prompts and edge cases.
• Implement fast evaluation checks before LLM-as-a-judge approaches.
• Create test suites for multi-turn agent conversations and safeguards.
• Design adversarial tests for prompt injection and security vulnerabilities.
• Build statistical evaluation frameworks for non-deterministic AI behavior.
• Integrate evaluation workflows into CI systems with SDK-level execution.
• Implement observability and tracing to monitor agent execution.
• Optimize evaluation efficiency using caching and mocking.
• Validate structured outputs and payload interactions with external systems.
• Evaluate performance under load and varying endpoint behavior.
• Independently own the evaluation architecture and processes.

Skills

Python

TypeScript

API testing

CI/CD

JSON schema

OpenTelemetry

LangSmith

Arize Phoenix

Weave

Tools

GitHub Actions

LangChain

LlamaIndex

AutoGen

CrewAI

Job description

Jobgether is seeking a founding AI Evaluation Engineer based in Mexico to design and build evaluation, testing, and observability infrastructure for production-grade AI agents. You will create scalable evaluation pipelines, regression testing, and adversarial scenarios to ensure reliability, safety, and efficiency across multi-turn conversations and API interactions.

As an independent contractor working remotely, you will own the evaluation architecture end-to-end, integrating with GitHub