Senior AI Evaluation Scientist - LLM Quality & Impact

Evolent; United States; Remote; USD 135,000 - 165,000; Full time

Job summary

Evolent seeks a data scientist to drive evaluation and improvement of LLM-based clinical services. You will own the diagnostic loop, design experiments, and certify fixes through structured evaluations, partnering with clinical reviewers and product managers to prioritize impactful failure modes.

You will mentor others, support guideline updates, and ensure compliant handling of PHI data while advancing observability and metrics for responsible AI in a health-tech setting.

Qualifications

• Bachelor’s degree or equivalent experience in a quantitative field.
• 5+ years of data science or applied ML experience with production models.
• 2+ years evaluating and improving LLM-based systems with structured error analysis.
• Demonstrated experience with LLM evaluation methods and tools.
• Strong Python and data-analysis skills; SQL a plus.
• Hypothesis-driven approach to isolate variables and prove fixes.
• Excellent written communication for engineers, clinicians, and leaders.

Responsibilities

• Own the diagnostic loop for LLM-based clinical services, form root-cause hypotheses, and design experiments.
• Test candidate fixes and verify improvements with structured evaluations.
• Own the evaluation roadmap and establish quality gates for deployments.
• Use and extend evaluation platforms, build golden sets and regression suites.
• Collaborate with clinical reviewers and product managers to prioritize failure modes.
• Mentor others and scale the evals function, including leadership readiness.
• Support annual clinical-guidelines updates with regression evaluation.
• Document the diagnostic playbook for broad reuse.
• Handle clinical data (including PHI) per security and compliance standards.

Skills

Data science

Machine learning

LLM evaluation

Python

SQL

Experiment design

Communication

Hypothesis-driven

Education

Bachelor's degree

Master's degree

Tools

OpenTelemetry

Langfuse

Logfire

Job description

Evolent seeks a data scientist to drive evaluation and improvement of LLM-based clinical services. You will own the diagnostic loop, design experiments, and certify fixes through structured evaluations, partnering with clinical reviewers and product managers to prioritize impactful failure modes.

You will mentor others, support guideline updates, and ensure compliant handling of PHI data while advancing observability and metrics for responsible AI in a health-tech setting.