Remote AI Evaluation Engineer — LLM Quality & Production

Sedgwick

Tallahassee (FL)

Remote

USD 120,000 - 180,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Sedgwick is seeking an AI Evaluation Engineer to design, build, and operate evaluation systems for GenAI solutions. You will support multiple product pods, establish evaluation standards, and create automated pipelines to measure LLM, RAG, and agented applications for production readiness.

You will work with product owners and SMEs, develop observability to monitor drift and latency, and design metrics and dashboards to inform leadership about quality and cost.

Qualifications

  • Master’s degree in computer science or a quantitative discipline preferred.
  • 4+ years in software, ML, or data engineering with hands-on GAI experience.
  • Strong Python skills and modern software engineering fundamentals.

Responsibilities

  • Establishes standardized, automated evaluation frameworks for LLMs and ML models.
  • Deploys observability to monitor drift, latency, and costs.
  • Define evaluation metrics for LLM/RAG/agent-based apps.
  • Build reusable evaluation frameworks and templates for pods.
  • Collaborate with product owners to translate requirements into quality criteria.
  • Develop CI/CD pipelines with regression tests and quality gates.
  • Implement offline/online evaluation and data governance practices.
  • Support A/B and shadow testing for model changes.

Skills

Python
CI/CD
Model evaluation
Observability tools
Communication
Team collaboration

Education

Master’s degree in CS or quantitative field

Tools

LangSmith
Arize Phoenix
WhyLabs
Cloud platforms (Azure/AWS/GCP)
MLflow
Eval tooling (Ragas/Langfuse)

Job description

Sedgwick is seeking an AI Evaluation Engineer to design, build, and operate evaluation systems for GenAI solutions. You will support multiple product pods, establish evaluation standards, and create automated pipelines to measure LLM, RAG, and agented applications for production readiness.

You will work with product owners and SMEs, develop observability to monitor drift and latency, and design metrics and dashboards to inform leadership about quality and cost.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Remote AI Evaluation Engineer - LLM Quality & CI/CD
Remote AI Evaluation Engineer - LLM Quality & CI/CD

Sedgwick • Orlando (FL)

Remote
USD 120,000 - 170,000
Remote AI Evaluation Engineer — LLM Quality & Safety
Remote AI Evaluation Engineer — LLM Quality & Safety

Sedgwick • Tallahassee (FL)

Remote
USD 120,000 - 190,000
Remote work
Career development
Remote AI Evaluation Engineer - LLM Quality & Safety
Remote AI Evaluation Engineer - LLM Quality & Safety

Sedgwick • Florida

Remote
USD 110,000 - 170,000
Remote AI Evaluation Engineer: LLM Quality & Risk
Remote AI Evaluation Engineer: LLM Quality & Risk

Sedgwick Claims Management Services Ltd • Florida

Hybrid
USD 110,000 - 170,000
AI Evaluation Engineer
AI Evaluation Engineer

Sedgwick • Tallahassee (FL)

Remote
USD 120,000 - 180,000
AI Evaluation Engineer
AI Evaluation Engineer

Sedgwick • Florida

Remote
USD 110,000 - 170,000
AI Evaluation Engineer
AI Evaluation Engineer

Sedgwick • Tallahassee (FL)

Remote
USD 120,000 - 190,000
Remote work
Career development
AI Evaluation Engineer
AI Evaluation Engineer

Sedgwick • Orlando (FL)

Remote
USD 120,000 - 170,000
AI Evaluation Engineer
AI Evaluation Engineer

Sedgwick Claims Management Services Inc. • Florida

Hybrid
USD 110,000 - 170,000
Evaluation AI Architect for Multi-Agent LLMs
Evaluation AI Architect for Multi-Agent LLMs

Not Disclosed • United States

Remote
USD 130,000 - 190,000