Senior Data Scientist, LLM Evaluation & BI

Paradigm Health; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

Paradigm Health is seeking a rigorous Data Scientist to measure and trust LLM-driven clinical data products and to build a reliable BI layer for the organization.

You will design production evaluations for LLM pipelines, establish a semantic layer, and implement production data models with SQL and dbt on Databricks, delivering measurable impact for patients and providers. You’ll collaborate with AI engineers, clinicians, and non-technical stakeholders to translate requirements into data products.

Qualifications

• Bachelor’s degree or higher in data science, statistics, biostatistics, computer science, mathematics, epidemiology, or a related field.
• 4+ years of experience as a data scientist, analytics engineer, ML/data professional, or other highly analytical role in life sciences, biotech, healthcare, or a related industry.
• Solid understanding of production LLM pipelines and hands-on experience evaluating LLM/NLP systems — accuracy metrics, error analysis, RAG/retrieval quality, prompt evaluation, etc.
• Strong proficiency in SQL, with experience building production data models in dbt (or a comparable transformation framework).
• Proficiency in at least one programming language (Python preferred; PySpark a plus).
• Experience with modern data platforms such as Databricks, and BI/analytics tools such as Hex, or similar.
• Fluency with data science best practices: reusability, version control, documentation, testing, code review, and reproducibility.
• Excellent written and verbal communication, with the ability to translate technical concepts into actionable insights for non-technical stakeholders and to develop reporting logic collaboratively with them.
• Comfort with AI tooling and agentic workflows in day‑to‑day work.
• Comfort with ambiguity and adaptability to a mission‑driven, fast‑paced startup environment.

Responsibilities

• Design and run evaluations of production LLM pipelines, developing accuracy and quality metrics that tell us how well our systems perform on real clinical tasks.
• Support evaluation of Patient Trial Evaluation (PTE) pipeline — assessing prompts and trial-matching logic for accuracy in reporting, and surfacing where and why they fail.
• Measure and improve the effectiveness of retrieval-augmented generation (RAG) systems, from retrieval quality through final output.
• Use accuracy metrics to inform cost/quality tradeoffs across trials, prompts, and models, giving the team a clear basis for which approaches to ship.
• Establish and maintain a library of “best prompts” for recurring clinical concepts, backed by evidence rather than intuition.
• Partner with AI Engineering to close the loop between evaluation findings and production improvements.
• Build and maintain production-grade data models using SQL and dbt on Databricks, ensuring analytics logic is reliable, maintainable, and production-ready.
• Help establish and grow a semantic layer that enables consistent, trusted reporting across the organization.
• Advance data governance practices across dbt, Databricks, and Hex — documentation, testing, versioning, and clear ownership of metrics.
• Collaborate with non-technical internal stakeholders to develop reporting logic, answer data questions, and translate business needs into well-defined data products.
• Support both internal and external reporting needs with accurate, well-tested datasets.
• For senior applicants: mentor and support junior data scientists — providing guidance on evaluation and analytics approaches, technical best practices, and professional growth.

Skills

SQL proficiency

Python

PySpark

Databricks

Hex BI

Data storytelling

Stakeholder communication

Education

Bachelor’s degree or higher in data science, statistics, biostatistics, computer science, mathematics, epidemiology, or related field

Tools

dbt

Databricks

Hex

Job description

Paradigm Health is seeking a rigorous Data Scientist to measure and trust LLM-driven clinical data products and to build a reliable BI layer for the organization.

You will design production evaluations for LLM pipelines, establish a semantic layer, and implement production data models with SQL and dbt on Databricks, delivering measurable impact for patients and providers. You’ll collaborate with AI engineers, clinicians, and non-technical stakeholders to translate requirements into data products.