Senior Evaluation AI Engineer (Fully Remote)

Not Disclosed; United States; Remote; USD 130,000 - 190,000; Full time

Job summary

Not Disclosed in the United States is seeking an AI Evaluation Engineer to design and implement automated evaluation pipelines for multi-agent LLM systems. The role focuses on building robust evaluation capabilities for our AI platform.

You will develop domain-specific metrics and grading approaches, create evaluation datasets, monitor system drift and observability, and support integration of evaluations into CI/CD for enterprise workflows such as lease abstraction and portfolio optimization.

Qualifications

• 5-7 years in software engineering, test automation, or AI engineering.
• Strong Python skills and experience with LangChain, LangGraph, and LangSmith.
• Experience with adversarial testing and AI safety evaluation.

Responsibilities

• Design and implement automated evaluation pipelines for multi-agent LLM systems.
• Develop custom metrics and grading approaches for domain-specific tasks.
• Build evaluation datasets and monitor system drift and observability.

Skills

Python

LangChain

LangGraph

LangSmith

AI engineering

Test automation

Tools

Ragas

DeepEval

Arize Phoenix

Job description

• Design and implement automated evaluation pipelines for multi-agent LLM systems.
• Develop custom metrics and grading approaches for domain-specific tasks.
• Build evaluation datasets and monitor system drift and observability.

Required Qualifications:

• 5-7 years in software engineering, test automation, or AI engineering.
• Strong Python skills and experience with LangChain, LangGraph, and LangSmith.
• Experience with adversarial testing and AI safety evaluation.

Preferred Qualifications:

• Experience with evaluation tools like Ragas, DeepEval, or Arize Phoenix.
• Familiarity with Bedrock Agents and integrating evaluations into CI/CD.

Not Disclosed

This company develops an AI-driven platform that automates complex enterprise workflows for corporate real estate, including lease abstraction and portfolio optimization. The size and culture of the organization are not specified in the posting.