Head of Data Science

Stealth Startup; United States; On-site; USD 180,000 - 260,000; Full time

Benefits offered by this job

Health benefits

Dental benefits

Job summary

Stealth Startup in the United States is seeking a Head of Data Science to lead and build our data science function. You will report to the CTO and collaborate closely with Engineering and Product to improve AI-powered clinical documentation systems.

This is a hands-on technical leadership role, owning evaluation frameworks, tooling, and data-driven improvements from the ground up as we scale our data science capabilities.

Qualifications

• 5+ years in data science, ML, or related field.
• Hands-on with LLM-based systems, evaluation, retrieval, embeddings, or fine-tuning.
• Strong programming skills and ability to ship production tools.
• Ability to translate ambiguous problems into concrete experiments and outcomes.
• Willingness to work within HIPAA constraints and healthcare data.

Responsibilities

• Design evaluation frameworks for AI pipelines, including automated and human assessments.
• Build data collection, labeling, and feedback loops from real-world usage.
• Develop experiment tools for prompts, models, retrieval strategies, and AI changes.
• Define model strategy, benchmarking, and appropriate AI approaches (prompting, PEFT/LoRA, etc.).
• Collaborate with engineers to deploy reliable data science-driven systems.

Skills

Python

Go

C#

Production‑grade tools

LLM evaluation

Experimental design

HIPAA knowledge

Healthcare domain

Job description

Benefits: Health, dental, and other employee benefits

About Us

We are building AI-powered software for clinical documentation and revenue cycle management. Our mission is to help physicians spend more time practicing medicine and less time managing EHR data entry and administrative workflows.

Our platform uses patient history, voice transcriptions from patient encounters, provider dictations, and proprietary and open datasets to generate clinical notes, assessment and plan orders, and billing codes for urology practices.

We are expanding into additional specialties and clinical workflows, including operative notes, clinical trial eligibility, and more advanced revenue cycle management capabilities.

We have built our product with a strong engineering team and modern LLM technology. We are now establishing our first dedicated Data Science function to create a rigorous approach to measuring, evaluating, and improving our AI systems.

About the Role

We are seeking a Head of Data Science to become our first dedicated Data Science hire. Reporting directly to the CTO, you will work closely with Engineering and Product to improve the accuracy, reliability, and measurability of our AI-powered systems.

This is a hands-on technical leadership role. You will work directly with data and models, write production-quality code, build evaluation and experimentation infrastructure, and investigate issues in production.

You will own the Data Science function from the ground up, defining its technical direction, establishing priorities, and determining the staffing and infrastructure needed as we grow.

Key Responsibilities

• Design and implement evaluation frameworks for our AI pipelines, including automated and human evaluation of clinical accuracy, completeness, consistency, and other domain-specific metrics.
• Build systems and processes for collecting, labeling, and incorporating provider feedback and real-world usage data into model and product improvements.
• Develop experimentation tools and processes for evaluating prompts, models, retrieval strategies, and other AI system changes.
• Define model and technical strategy, including selecting and benchmarking models and determining when prompting, retrieval, fine-tuning, PEFT/LoRA, preference tuning, or other approaches are appropriate.
• Combine LLMs with classical ML and statistical techniques such as entity extraction, assertion and negation detection, embeddings, and retrieval when they improve accuracy, cost, latency, or reliability.
• Partner with engineers to build reliable production systems incorporating data science and ML techniques, including structured outputs, confidence scoring, and validation layers.
• Establish practical development standards for reproducible experiments, model evaluation, monitoring, and production reliability.
• Investigate production issues and identify data-driven opportunities to improve AI system performance.
• Partner with Product to identify high-value opportunities for applying data science and machine learning to new clinical workflows and specialties.
• Define the future structure, priorities, and staffing needs of the Data Science function as we scale.

Required Qualifications

• 5+ years of experience in data science, machine learning, applied ML, or a related field.
• Strong programming skills in Python, Go, C#, or a similar language.
• Experience building production-quality tools and systems rather than only notebooks and prototypes.
• Experience designing and implementing evaluation frameworks for ML or AI systems, including generative AI systems where correctness is difficult to measure using simple automated metrics.
• Hands‑on experience with LLM-based systems, including model evaluation, experimentation, retrieval, embeddings, or fine‑tuning.
• Strong understanding of statistical methods, experimental design, and quantitative analysis.
• Ability to turn ambiguous product or technical problems into concrete experiments, implementations, and measurable outcomes.
• Ability to work independently as the first specialist establishing a new technical discipline.
• Willingness and ability to work within HIPAA requirements and the constraints associated with protected health information.
• Ability to quickly learn and operate within the healthcare domain.

Nice to Have

• Experience with clinical NLP, medical coding, or healthcare revenue cycle workflows.
• Experience working with clinical, healthcare, regulated, or other sensitive data.
• Experience working with structured and unstructured healthcare data.
• Experience applying AI/ML to clinical or healthcare workflows.

What Success Looks Like

Within the first six months, you will establish a rigorous, measurable approach to evaluating and improving our AI systems.

Success will include:

• Building an automated evaluation suite that can gate releases of our AI pipelines.
• Establishing a clinician feedback loop that turns real‑world feedback, edits, and corrections into measurable quality improvements.
• Creating systematic experimentation processes for models, prompts, retrieval strategies, and other AI changes.
• Providing data‑driven insight into what works, what does not, and why.
• Defining the next priorities, infrastructure, and staffing requirements for our Data Science function.

You will serve as the technical owner of Data Science while remaining deeply involved in hands‑on implementation and day‑to‑day technical work.