Remote AI Evaluation Engineer - LLM Quality & Safety

Sedgwick; Florida; Remote; USD 110,000 - 170,000; Full time

Job summary

Sedgwick is seeking an AI Evaluation Engineer to design, build, and operate evaluation systems for GenAI solutions. You will establish consistent evaluation standards, automated pipelines, and quality gates across product pods, supporting LLM, RAG, and agent-based apps.

The role is fully remote for US-based candidates. Ideal candidates have 4+ years in ML/software, strong Python skills, and experience with evaluation tools and cloud platforms.

Qualifications

• Master's degree in CS or a quantitative discipline.
• 4+ years in software, ML, or data engineering with 1–2 years on LLM-based apps.
• Experience with evaluation and observability frameworks.
• Strong Python skills and software engineering fundamentals.
• Experience with LLM/RAG systems, retrieval metrics and generation metrics.
• Familiarity with AI risk, governance frameworks, and secure development practices.

Responsibilities

• Establish automated evaluation frameworks for LLMs and ML models.
• Deploy observability to monitor drift, latency, and costs.
• Define evaluation metrics for LLM, RAG, and agent-based apps.
• Build reusable evaluation templates and libraries for pods.
• Translate business requirements into measurable quality criteria.
• Develop CI/CD pipelines with regression tests and quality gates.
• Implement offline and online evaluation, including dashboards.
• Support A/B and shadow testing for model and prompt changes.
• Design LLM-as-a-judge evaluators and calibration with human labels.
• Create human annotation workflows and inter-annotator agreement.

Skills

Evaluation frameworks

Python

Statistical validation

LLM/RAG evaluation

Observability tools

Cloud platforms

Communication

Team collaboration

Model risk & safety

Education

Master's degree in CS or quantitative discipline

Tools

LangSmith

Arize Phoenix

WhyLabs

Ragas

DeepEval

promptfoo

Langfuse

MLflow

Job description

Sedgwick is seeking an AI Evaluation Engineer to design, build, and operate evaluation systems for GenAI solutions. You will establish consistent evaluation standards, automated pipelines, and quality gates across product pods, supporting LLM, RAG, and agent-based apps.

The role is fully remote for US-based candidates. Ideal candidates have 4+ years in ML/software, strong Python skills, and experience with evaluation tools and cloud platforms.