Remote AI Evaluation Engineer — LLM Quality & Safety

Sedgwick; Tallahassee (FL); Remote; USD 120,000 - 190,000; Full time

Benefits offered by this job

Remote work

Career development

Job summary

Sedigwick is seeking an AI Evaluation Engineer to design, build, and operate evaluation systems for GenAI solutions in production. You will establish standardized evaluation standards, automate pipelines, and ensure quality across LLM, RAG, and agent-based applications.

The role is fully remote for U.S.-based candidates and offers collaboration with product teams, SMEs, and risk stakeholders, with emphasis on observability, testing, and CI/CD integration.

Qualifications

• Requires a Master’s degree in a computer science or quantitative field.
• 4+ years in software, ML, or data engineering with hands-on with LLMs.
• Strong Python skills, testing, CI/CD, and model evaluation metrics.

Responsibilities

• Establish standardized, automated evaluation frameworks for LLMs and ML models.
• Deploy observability solutions to monitor drift, latency, and cost of deployments.
• Define evaluation metrics for LLM/RAG/agent applications and create reusable eval templates.
• Develop CI/CD pipelines for evaluation, including offline and online testing.
• Collaborate with product owners and SMEs to translate requirements into quality criteria.

Skills

Python

CI/CD

LangSmith

MLOps

Education

Master’s degree in computer science or quantitative discipline

Tools

LangSmith

Arize Phoenix

WhyLabs

MLflow

Job description

Sedigwick is seeking an AI Evaluation Engineer to design, build, and operate evaluation systems for GenAI solutions in production. You will establish standardized evaluation standards, automate pipelines, and ensure quality across LLM, RAG, and agent-based applications.

The role is fully remote for U.S.-based candidates and offers collaboration with product teams, SMEs, and risk stakeholders, with emphasis on observability, testing, and CI/CD integration.