Remote AI Evaluation Engineer - LLM Quality & Metrics

Sedgwick Claims Management Services Ltd; Florida; Remote; USD 120,000 - 180,000; Full time

Job summary

Sedgwick Claims Management Services Ltd is seeking an AI Evaluation Engineer for a fully remote role open to U.S. based candidates. You will design evaluation systems for GenAI solutions, build CI/CD pipelines, and monitor model performance across pods.

You will collaborate with data scientists and engineers to ensure evaluation quality, reliability, and readiness for production, enabling safe, responsible AI use.

Qualifications

• Master’s degree in CS or quantitative discipline preferred.
• 4+ years in software, ML, or data engineering with 1–2 years on LLM-based apps.
• Experience with evaluation and observability frameworks.
• Familiarity with cloud LLM services.

Responsibilities

• Establishes automated evaluation frameworks for LLMs and ML models.
• Deploys observability to monitor drift, latency, and resource use.
• Defines evaluation metrics for LLM, RAG, and agent-based apps.
• Builds reusable eval templates for pods to adopt.
• Collaborates with product owners to translate requirements into quality criteria.
• Develops CI/CD pipelines with regression tests and gates.

Skills

Python

CI/CD fundamentals

Statistical validation

Experiment design

Communication

Education

Master's degree in computer science or quantitative discipline

Tools

LangSmith

Arize Phoenix

WhyLabs

MLflow

Job description

Sedgwick Claims Management Services Ltd is seeking an AI Evaluation Engineer for a fully remote role open to U.S. based candidates. You will design evaluation systems for GenAI solutions, build CI/CD pipelines, and monitor model performance across pods.

You will collaborate with data scientists and engineers to ensure evaluation quality, reliability, and readiness for production, enabling safe, responsible AI use.