Evaluation AI Architect for Multi-Agent LLMs

Not Disclosed; United States; Remote; USD 130,000 - 190,000; Full time

Job summary

Not Disclosed in the United States is seeking an AI Evaluation Engineer to design and implement automated evaluation pipelines for multi-agent LLM systems. The role focuses on building robust evaluation capabilities for our AI platform.

You will develop domain-specific metrics and grading approaches, create evaluation datasets, monitor system drift and observability, and support integration of evaluations into CI/CD for enterprise workflows such as lease abstraction and portfolio optimization.

Qualifications

• 5-7 years in software engineering, test automation, or AI engineering.
• Strong Python skills and experience with LangChain, LangGraph, and LangSmith.
• Experience with adversarial testing and AI safety evaluation.

Responsibilities

• Design and implement automated evaluation pipelines for multi-agent LLM systems.
• Develop custom metrics and grading approaches for domain-specific tasks.
• Build evaluation datasets and monitor system drift and observability.

Skills

Python

LangChain

LangGraph

LangSmith

AI engineering

Test automation

Tools

Ragas

DeepEval

Arize Phoenix

Job description

Not Disclosed in the United States is seeking an AI Evaluation Engineer to design and implement automated evaluation pipelines for multi-agent LLM systems. The role focuses on building robust evaluation capabilities for our AI platform.

You will develop domain-specific metrics and grading approaches, create evaluation datasets, monitor system drift and observability, and support integration of evaluations into CI/CD for enterprise workflows such as lease abstraction and portfolio optimization.