Remote Senior AI Evaluation Engineer

LTM; Irving (TX); Remote; USD 120,000 - 180,000; Full time

Benefits offered by this job

Term and Long-Term Disability Coverage

Plan with Company match.

Paternity and Maternity Leave

Job summary

LTM is seeking a Senior AI Evaluation Engineer to design and operate end-to-end evaluation frameworks for agentic AI systems. The role focuses on offline and online testing, rubric design, and multi-agent evaluation pipelines, enabling safer and more grounded AI behavior.

The ideal candidate will have hands-on production-grade experience with GenAI, LLM-as-judge proficiency, and strong knowledge of evaluation metrics.

Qualifications

• GenAI experience at an advanced level with Python and production-grade systems.
• LLM-as-judge rubric design and scoring at session, response and tool-call levels.
• Agent evaluation experience: offline/on-demand regression testing and online live-traffic scoring.
• Agentic framework experience (Strands, LangGraph, CrewAI or similar).
• Experience with AWS Bedrock, CloudWatch; Lambda, S3, IAM fundamentals.

Responsibilities

• Design and operate end-to-end evaluation frameworks for agentic AI systems (offline regression, online monitoring, A/B testing).
• Define quality rubrics, scoring architectures, and evaluation standards across multi-agent workflows.
• Build LLM-as-judge and Agent-as-judge pipelines assessing trajectory, tool-call accuracy, groundedness, and safety.

Skills

GenAI Experience

Advanced Python

Production-grade systems

Agent evaluation

LLM-as-judge proficiency

Evaluation metrics

Agentic framework experience

AWS Bedrock

CloudWatch

Lambda

S3

IAM fundamentals

Tools

AWS Bedrock

CloudWatch

Lambda

S3

IAM

Job description

LTM is seeking a Senior AI Evaluation Engineer to design and operate end-to-end evaluation frameworks for agentic AI systems. The role focuses on offline and online testing, rubric design, and multi-agent evaluation pipelines, enabling safer and more grounded AI behavior.

The ideal candidate will have hands-on production-grade experience with GenAI, LLM-as-judge proficiency, and strong knowledge of evaluation metrics.