Remote AI Evaluation Engineer — LLM Quality & Safety

Sedgwick; New York (NY); Remote; USD 120,000 - 180,000; Full time

Benefits offered by this job

Fully remote

Work-life balance

Job summary

Sedgwick is hiring an AI Evaluation Engineer to design and operate evaluation systems for GenAI solutions. You will establish standardized frameworks, automate pipelines, and monitor model quality across products, including LLM, RAG, and agent-based apps.

The role is fully remote for U.S.-based candidates, with focus on reducing risk, improving reliability, and ensuring readiness for production through metrics, dashboards, and collaboration with product teams.

Qualifications

• Master’s degree preferred in CS or quantitative field
• 4+ years in software, ML, or data engineering
• Hands-on with LLM-based applications (1–2 years)
• Proficient with evaluation/observability tools and metrics
• Strong Python and CI/CD fundamentals

Responsibilities

• Establish standardized evaluation frameworks for LLMs and ML models
• Deploy observability solutions to monitor drift, latency, and cost
• Define evaluation metrics for LLM/RAG/agent apps and validate results
• Build reusable evaluation templates and CI/CD pipelines for eval changes
• Coordinate with product owners to translate requirements into quality criteria
• Produce evaluation evidence for security, privacy, and risk reviews

Skills

Python

CI/CD

Data analysis

Collaboration

Education

Master’s degree in CS or quantitative field

Tools

LangSmith

Arize Phoenix

WhyLabs

Langfuse

MLflow

Job description

Sedgwick is hiring an AI Evaluation Engineer to design and operate evaluation systems for GenAI solutions. You will establish standardized frameworks, automate pipelines, and monitor model quality across products, including LLM, RAG, and agent-based apps.

The role is fully remote for U.S.-based candidates, with focus on reducing risk, improving reliability, and ensuring readiness for production through metrics, dashboards, and collaboration with product teams.