Senior AI Quality & Evaluation Scientist — Remote

Matcha; Northern (KY); Hybrid; USD 180,000 - 230,000; Full time

Job summary

Ellipsis Health is seeking a Data Scientist specializing in AI Quality & Evaluation to ensure the robustness, accuracy, and fairness of our AI models and systems. You will build evaluation frameworks, curate diverse datasets, and drive continuous improvement across our AI initiatives to accelerate customer onboarding and enhance user experience.

You will collaborate with AI/ML engineers and product managers, mentor junior data scientists, and stay current with LLM and speech evaluation methods

Qualifications

• 5+ years of industry experience in data science or ML.
• BS or MS in Computer Science, Data Science, Statistics or Mathematics.
• Proficiency in Python and libraries such as Pydantic, Pandas, NumPy, Scikit-learn.
• Strong statistics, experimental design, hypothesis testing, and A/B testing.
• Experience with large-scale datasets, data pipelines, and analytics tools.
• Familiarity with evaluation metrics like precision, recall, and Cohen's Kappa.

Responsibilities

• Own data science and AI evaluation projects end-to-end from problem to implementation.
• Develop data and evaluation infrastructure for a closed-loop AI quality system.
• Design and implement robust evaluation frameworks, methodologies, and metrics.
• Build and maintain high-quality datasets for training, testing, and evaluation.
• Create and manage scalable data pipelines and dashboards for analysis.
• Collaborate with AI/ML Engineers and Product Managers to drive product improvements.
• Mentor junior data scientists and raise the technical bar.

Skills

Python

Statistical analysis

A/B testing

Communication skills

Education

Bachelor's or Master's in Computer Science, Data Science, Statistics or Mathematics

Tools

Pydantic

Pandas

NumPy

Scikit-learn

Job description

Ellipsis Health is seeking a Data Scientist specializing in AI Quality & Evaluation to ensure the robustness, accuracy, and fairness of our AI models and systems. You will build evaluation frameworks, curate diverse datasets, and drive continuous improvement across our AI initiatives to accelerate customer onboarding and enhance user experience.

You will collaborate with AI/ML engineers and product managers, mentor junior data scientists, and stay current with LLM and speech evaluation methods