Senior Data Scientist: LLM Eval & BI for Trials

Paradigm Health, Inc.; Town of Columbus (NY); Hybrid; USD 120,000 - 180,000; Full time

Job summary

Paradigm Health, Inc. is seeking a rigorous Data Scientist to define how we measure and trust LLM-powered clinical data products and to build a reliable business intelligence layer across the org.

You'll work at the intersection of AI evaluation, analytics engineering, and clinical research, partnering with AI engineers, data engineers, clinicians, and business stakeholders to translate questions into well-defined data products.

Qualifications

• Bachelor's degree or higher in data science or related field.
• 4+ years as a data scientist or analytics role in life sciences or healthcare.
• Experience evaluating LLM/NLP systems and performance metrics.
• Strong SQL and dbt production data modeling experience.
• Python (preferred) and PySpark proficiency.
• Experience with Databricks and BI tools like Hex.

Responsibilities

• Design and run evaluations of production LLM pipelines with accuracy metrics.
• Evaluate Patient Trial Evaluation (PTE) pipeline for accuracy and failure roots.
• Improve retrieval-augmented generation (RAG) systems end-to-end.
• Inform cost/quality tradeoffs across trials, prompts, and models.
• Create a library of best prompts for clinical concepts with evidence.
• Collaborate with AI Engineering to bridge evaluation and production.
• Build and maintain data models using SQL and dbt on Databricks.
• Develop a semantic layer for consistent, trusted reporting.
• Advance data governance across dbt, Databricks, and Hex.
• Translate business needs into well-defined data products with non-technical stakeholders.
• Support internal and external reporting with well-tested datasets.
• Mentor and support junior data scientists for senior applicants.

Education

Bachelor's degree or higher in data science, statistics, biostatistics, computer science, mathematics, epidemiology, or a related field

Tools

Databricks

dbt

Hex

PySpark

SQL

Job description

Paradigm Health, Inc. is seeking a rigorous Data Scientist to define how we measure and trust LLM-powered clinical data products and to build a reliable business intelligence layer across the org.

You'll work at the intersection of AI evaluation, analytics engineering, and clinical research, partnering with AI engineers, data engineers, clinicians, and business stakeholders to translate questions into well-defined data products.