Remote AI Evaluation Engineer - LLM Quality & CI/CD

Sedgwick; Orlando (FL); Remote; USD 120,000 - 170,000; Full time

Job summary

Sedgwick is seeking an AI Evaluation Engineer to design, build, and operate evaluation systems for GenAI solutions. You will establish standardized evaluation frameworks, automated pipelines, and quality gates across LLM, RAG, and agentic applications to ensure accuracy, safety, and production readiness.

In this remote role, you will collaborate with product owners and SMEs to translate business needs into measurable criteria, develop CI/CD pipelines, monitor model performance, and drive

Qualifications

• Master’s degree in computer science or a quantitative discipline preferred.
• 4+ years in software, ML, or data engineering with hands-on work on LLM-based applications.

Responsibilities

• Establishes automated evaluation frameworks for LLMs and traditional ML models.
• Deploys observability to monitor drift, latency, and cost.
• Defines evaluation metrics for LLM, RAG, and agent-based apps.
• Builds reusable evaluation pipelines and libraries for pods.
• Collaborates with product owners to translate requirements into measurable quality criteria.
• Implements CI/CD pipelines with automated testing and quality gates.
• Develops offline/online evaluation methods including dashboards and reporting.
• Supports A/B and shadow testing for model changes.
• Designs LLM-as-a-judge evaluators and calibration with human labels.

Skills

Evaluation frameworks

Observability

Python

ML evaluation

CI/CD

Cross-functional collaboration

Education

Master’s degree in computer science or quantitative field

Tools

LangSmith

Langfuse

MLflow

Job description

Sedgwick is seeking an AI Evaluation Engineer to design, build, and operate evaluation systems for GenAI solutions. You will establish standardized evaluation frameworks, automated pipelines, and quality gates across LLM, RAG, and agentic applications to ensure accuracy, safety, and production readiness.

In this remote role, you will collaborate with product owners and SMEs to translate business needs into measurable criteria, develop CI/CD pipelines, monitor model performance, and drive