Senior Data Scientist, Applied AI — LLM/RAG Apps

definitivehc; United States; On-site; USD 136,000 - 206,000; Full time

Job summary

Definitive Healthcare, headquartered near Boston, seeks a Senior Data Scientist to own AI evaluation for healthcare-focused use cases. You will build and optimize LLM, RAG, and tool-using workflows in Python, leveraging Databricks to deliver reliable, scalable AI solutions.

You will evaluate models, embeddings, and retrieval methods, document tradeoffs, and collaborate with engineering and product teams to ensure high quality and cost-conscious deployments.

Qualifications

• Bachelor's degree required; master's degree in a quantitative or technical field preferred.
• 5+ years of relevant experience in data science, machine learning, applied AI, software engineering, or a related field.
• Hands-on experience building and evaluating LLM applications.
• Strong Python programming, software engineering and testing, and analytical problem-solving skills.
• Strong knowledge of LLMs, embeddings, semantic search, retrieval systems, and RAG architectures.
• Experience with Databricks, including Delta Lake, Unity Catalog, Lakebase, Lakeflow Jobs, and MLflow.
• Ability to design representative evaluations for an assigned use case, analyze errors, and use results to improve a system.
• Ability to weigh quality, reliability, latency, and cost when recommending an approach for their own project.
• Clear communication and effective collaboration with technical and nontechnical partners.
• Experience with agent frameworks and AI evaluation platforms.
• Experience with healthcare data is preferred, but not required.

Responsibilities

• Own assigned workflows: build, test, and improve LLM, RAG, and tool-using agent workflows with maintainable Python code.
• Apply the evaluation framework by selecting criteria, baselines, and test datasets.
• Select technical approaches by comparing models, embeddings, and retrieval methods with evidence on quality, latency, and cost.
• Diagnose and resolve issues: investigate retrieval misses, incorrect answers, and workflow failures, and implement fixes.
• Maintain and extend test assets: create test cases, benchmark datasets, and regression checks.
• Support production reliability: monitor performance and implement improvements.
• Collaborate with engineering, product, and domain partners to communicate findings and tradeoffs.

Skills

Python programming

Software engineering

Testing

Analytical problem-solving

LLMs

Embeddings

Semantic search

Retrieval systems

RAG architectures

Agent frameworks

AI evaluation platforms

Healthcare data

Education

Bachelor's degree

Master's degree

Tools

Databricks

Delta Lake

Unity Catalog

Lakebase

Lakeflow Jobs

MLflow

Job description

Definitive Healthcare, headquartered near Boston, seeks a Senior Data Scientist to own AI evaluation for healthcare-focused use cases. You will build and optimize LLM, RAG, and tool-using workflows in Python, leveraging Databricks to deliver reliable, scalable AI solutions.

You will evaluate models, embeddings, and retrieval methods, document tradeoffs, and collaborate with engineering and product teams to ensure high quality and cost-conscious deployments.