Founding AI Evaluation Architect (Remote)

Jobgether; Italia; Teletrabalho; EUR 83 000 - 165 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Fully remote work

Negotiable compensation

Independent contractor arrangement

Flexible working environment

Professional development

Resumo da oferta

Jobgether is partnering to hire an AI Evaluation Engineer based in Italy. You will design and build the evaluation, testing, and observability infrastructure to ensure AI agents perform reliably, safely, and efficiently.

You will develop automated tests for complex reasoning, multi-turn conversations, and API interactions, while implementing regression testing, adversarial scenarios, and scalable pipelines. This is a fully remote, globally distributed role with ownership from the ground up.

Qualificações

• Strong Python or TypeScript skills for API testing and automation.
• Experience in building LLM evaluation pipelines and AI testing frameworks.
• Proven CI/CD integration, especially GitHub Actions, with SDK-level execution.
• Validation of JSON schemas, OpenAPI specs, and function-calling payloads.
• Familiarity with observability tools like OpenTelemetry and LangSmith.
• Ability to design and own an evaluation architecture from scratch.
• Excellent English communication and overlap with US Central Time.
• Experience with code execution sandboxes or AI agent frameworks.

Responsabilidades

• Design, build, and maintain automated CI/CD testing pipelines.
• Develop golden datasets for regression prompts and edge cases.
• Implement fast deterministic checks including schema validation and API status checks.
• Create test suites for multi-turn agent conversations and tool use.
• Design adversarial tests for prompt injection and security risks.
• Build statistical evaluation frameworks for non-deterministic AI behavior.
• Integrate evaluation workflows into GitHub Actions and CI systems.
• Implement observability and tracing with OpenTelemetry, LangSmith, Arize.
• Optimize evaluation cost via caching and mocking.
• Validate outputs like JSON schemas and OpenAPI payloads.
• Evaluate performance under load and varied latency.
• Own the evaluation architecture and improve testing standards.

Ferramentas

Python

TypeScript

CI/CD

API Testing

Automated Testing

Observability

OpenTelemetry

LangSmith

Arize Phoenix

JSON Schema

OpenAPI

LangChain

LlamaIndex

AutoGen

CrewAI

GitHub Actions

Descrição da oferta de emprego

Jobgether is partnering to hire an AI Evaluation Engineer based in Italy. You will design and build the evaluation, testing, and observability infrastructure to ensure AI agents perform reliably, safely, and efficiently.

You will develop automated tests for complex reasoning, multi-turn conversations, and API interactions, while implementing regression testing, adversarial scenarios, and scalable pipelines. This is a fully remote, globally distributed role with ownership from the ground up.