Remote AI Evaluation Engineer: LLM Quality & Risk

Sedgwick Claims Management Services Ltd; Florida; Hybrid; USD 110,000 - 170,000; Full time

Job summary

Sedgwick is seeking an AI Evaluation Engineer to design, build, and operate evaluation systems for GenAI solutions. You will standardize evaluation processes across product pods, implement automated pipelines, and ensure quality gates for prompt, model, and retrieval changes.

The role emphasizes observability, model risk reporting, and collaboration with product owners and SMEs to translate requirements into measurable criteria. Remote-first within the United States.

Qualifications

• Master’s degree in a computer science or quantitative discipline preferred.
• 4+ years in software, ML, or data engineering with 1–2 years hands-on with LLM-based apps.
• Experience with evaluation/observability tools like LangSmith, Arize Phoenix, WhyLabs.

Responsibilities

• Establishes automated evaluation frameworks for LLMs and traditional ML models.
• Deploys observability solutions to monitor drift, latency, and resource usage.
• Define evaluation methodologies and metrics for LLM, RAG, and agent-based applications.
• Build reusable evaluation templates and libraries for pods.

Skills

Python

CI/CD

LLM evaluation

Observability

LangSmith

Education

Master's degree in CS/quantitative field

Tools

Azure

AWS

GCP

Job description

Sedgwick is seeking an AI Evaluation Engineer to design, build, and operate evaluation systems for GenAI solutions. You will standardize evaluation processes across product pods, implement automated pipelines, and ensure quality gates for prompt, model, and retrieval changes.

The role emphasizes observability, model risk reporting, and collaboration with product owners and SMEs to translate requirements into measurable criteria. Remote-first within the United States.