Data Scientist (AI Quality & Evaluation)

Bioscope.ai, Inc.; Boston (MA); On-site; USD 100,000 - 130,000; Full time

Job summary

A leading clinical AI company is seeking a Data Scientist responsible for ensuring the quality and reliability of AI outputs. You will design automated evaluation pipelines, develop uncertainty quantification systems, and build comprehensive frameworks that combine automated assessments with clinician validations. The ideal candidate has expertise in deep learning frameworks such as PyTorch, experience with model evaluation metrics, and excellent communication skills. A Master's degree in a related field is preferred.

Qualifications

• Strong foundation in deep learning frameworks (PyTorch) and LLM architectures.
• Experience with model evaluation, benchmarking, and quality metrics.
• Proficiency in Python and modern ML development tools.

Responsibilities

• Design and implement automated evaluation pipelines to assess AI output quality.
• Develop uncertainty quantification systems correlating confidence scores with accuracy.
• Build evaluation frameworks combining automated assessment and clinician-validated cases.

Skills

Deep learning frameworks (PyTorch)

Model evaluation and quality metrics

Proficiency in Python

Statistical analysis

Communication skills

Education

Master's degree in Computer Science, Machine Learning, Statistics, or related field

Job description

About the Role

We're looking for a Data Scientist to own the quality, reliability, and trustworthiness of our clinical AI outputs. You'll build the systems that ensure our AI "knows what it doesn't know"—developing evaluation frameworks, calibrated confidence scoring, and automated quality assurance that physicians can actually trust.

What You'll Do

• Design and implement automated evaluation pipelines that assess AI output quality, accuracy, and safety at scale

• Develop uncertainty quantification systems where confidence scores meaningfully correlate with accuracy

• Build comprehensive evaluation frameworks combining automated assessment with clinician-validated test cases

• Implement feedback loops that continuously improve model outputs based on validation signals

• Establish scalable quality gates that catch errors before they reach end users

• Contribute to model alignment and fine-tuning efforts

Qualifications

Required

• Strong foundation in deep learning frameworks (PyTorch) and LLM architectures

• Experience with model evaluation, benchmarking, and quality metrics

• Proficiency in Python and modern ML development tools

• Strong statistical foundations

• Ability to read, implement, and extend research papers

• Excellent communication skills

Preferred

• Master's degree in Computer Science, Machine Learning, Statistics, or related quantitative field (PhD preferred)

• Publications in top ML/AI venues (NeurIPS, ICML, ICLR, ACL)

• Experience with RLHF, DPO, or preference optimization techniques

• Background in healthcare AI or regulated industries

• Experience building evaluation systems for production LLM applications