AI Evaluation & Improvement Lead for LLMs

Evolent Health LLC; United States; Remote; USD 135,000 - 165,000; Full time

Benefits offered by this job

Health insurance

Comprehensive benefits

Job summary

Evolent Health LLC is seeking a Senior Data Scientist to own the diagnostic loop for LLM-based clinical services. You will design experiments, evaluate fixes, and define the evaluation roadmap with clinicians and product managers.

Responsibilities include building and extending evaluation platforms, creating golden sets and regression suites, and mentoring a growing team of data scientists while ensuring compliance with privacy and security standards.

Qualifications

• Bachelor’s degree in data science, CS, statistics or related quantitative field or equivalent experience.
• 5+ years of data science or applied ML experience including shipping models in production.
• 2+ years evaluating and improving LLM-based systems with structured error analysis, prompt/configuration iteration, and experiment design.
• Demonstrated experience with LLM evaluation methods and tools such as golden/regression sets and LLM-as-judge with validation.
• Strong Python and data-analysis skills; SQL a plus; ability to reason about metrics like sensitivity, specificity, and PPV.
• Hypothesis-driven mindset with ability to isolate variables and prove fixes.
• Strong written communication; able to explain findings to engineers, clinicians, and leadership.

Responsibilities

• Own the diagnostic loop for LLM-based clinical services and verify fixes with structured evaluations.
• Own the evaluation roadmap and quality bar for the team’s AI services and governance.
• Build and extend evaluation platforms, golden sets, and regression suites; define deployment gates.
• Mentor data scientists and help scale the evals function as the team grows.
• Collaborate with clinical reviewers and product managers on evidence-based criteria; ensure privacy and compliance.

Skills

LLM evaluation

Python

SQL

Experiment design

Structured error analysis

Metrics interpretation

Communication

Hypothesis-driven

Production ML systems

Data analysis

Education

Bachelor's degree in Data Science/CS/Statistics

Master's degree in a quantitative field

Tools

OpenTelemetry

Logfire

Langfuse

Job description

Evolent Health LLC is seeking a Senior Data Scientist to own the diagnostic loop for LLM-based clinical services. You will design experiments, evaluate fixes, and define the evaluation roadmap with clinicians and product managers.

Responsibilities include building and extending evaluation platforms, creating golden sets and regression suites, and mentoring a growing team of data scientists while ensuring compliance with privacy and security standards.