Remote AI Evaluation Engineer - LLM & Quality Assurance

Sedgwick; Oregon; Hybrid; USD 130,000 - 185,000; Full time

Job summary

Sedgwick is seeking an AI Evaluation Engineer to design and operate evaluation systems for GenAI solutions. You will standardize evaluation practices, implement automated pipelines, and ensure quality before production releases.

The role is fully remote for U.S.-based candidates and focuses on LLM, RAG, and agentic applications. You will partner with pod teams to translate business needs into measurable quality criteria, deploy observability to monitor model drift and safety, and contribute to

Qualifications

• Master’s degree in computer science or quantitative field preferred.
• 4+ years in software, ML, or data engineering with 1–2 years in LLM‑based apps.
• Hands-on with evaluation and observability tooling and metrics.

Responsibilities

• Establishes automated evaluation frameworks for LLMs and ML models.
• Develops CI/CD pipelines with regression tests and quality gates.
• Defines evaluation metrics for LLM, RAG, and agent apps.
• Builds reusable eval templates and libraries for pods.
• Collaborates with product owners to set measurable quality criteria.
• Monitors drift, latency, and cost; optimizes for security and safety.

Skills

Python programming

LLM evaluation

Observability frameworks

Prompt engineering

Statistics & experimental design

Cloud platforms (Azure/AWS/GCP)

Communication skills

Team collaboration

LLM-as-a-judge design

Education

Master’s degree in CS or quantitative discipline

Tools

LangSmith

Arize Phoenix

WhyLabs

MLflow

Job description

Sedgwick is seeking an AI Evaluation Engineer to design and operate evaluation systems for GenAI solutions. You will standardize evaluation practices, implement automated pipelines, and ensure quality before production releases.

The role is fully remote for U.S.-based candidates and focuses on LLM, RAG, and agentic applications. You will partner with pod teams to translate business needs into measurable quality criteria, deploy observability to monitor model drift and safety, and contribute to