Head of Data Science & AI Systems

Stealth Startup; United States; On-site; USD 180,000 - 270,000; Full time

Benefits offered by this job

Health, dental, and other employee ben

Job summary

Stealth Startup is seeking a Head of Data Science in the United States to lead and build our first dedicated Data Science function. You will work directly with the CTO and cross-functionally with Engineering and Product to improve AI systems, design evaluation frameworks, and guide production-ready ML improvements.

You will own the data science function, set technical direction, and determine staffing and infrastructure needs as we scale, while remaining hands-on in implementation and model

Qualifications

• 5+ years of experience in data science, machine learning, applied ML, or a related field.
• Strong programming skills in Python, Go, C#, or a similar language.
• Experience building production-quality tools and systems rather than only notebooks and prototypes.
• Experience designing and implementing evaluation frameworks for ML or AI systems, including generative AI systems where correctness is difficult to measure using simple automated metrics.
• Hands-on experience with LLM-based systems, including model evaluation, experimentation, retrieval, embeddings, or fine-tuning.
• Strong understanding of statistical methods, experimental design, and quantitative analysis.
• Ability to turn ambiguous product or technical problems into concrete experiments, implementations, and measurable outcomes.
• Ability to work independently as the first specialist establishing a new technical discipline.
• Willingness and ability to work within HIPAA requirements and the constraints associated with protected health information.
• Ability to quickly learn and operate within the healthcare domain.

Responsibilities

• Design and implement evaluation frameworks for our AI pipelines, including automated and human evaluation of clinical accuracy, completeness, consistency, and other domain-specific metrics.
• Build systems and processes for collecting, labeling, and incorporating provider feedback and real-world usage data into model and product improvements.
• Develop experimentation tools and processes for evaluating prompts, models, retrieval strategies, and other AI system changes.
• Define model and technical strategy, including selecting and benchmarking models and determining when prompting, retrieval, fine-tuning, PEFT/LoRA, preference tuning, or other approaches are appropriate.
• Combine LLMs with classical ML and statistical techniques such as entity extraction, assertion and negation detection, embeddings, and retrieval when they improve accuracy, cost, latency, or reliability.
• Partner with engineers to build reliable production systems incorporating data science and ML techniques, including structured outputs, confidence scoring, and validation layers.
• Establish practical development standards for reproducible experiments, model evaluation, monitoring, and production reliability.
• Investigate production issues and identify data-driven opportunities to improve AI system performance.
• Partner with Product to identify high-value opportunities for applying data science and machine learning to new clinical workflows and specialties.
• Define the future structure, priorities, and staffing needs of the Data Science function as we scale.

Skills

Python

Go

C#

Production tools

LLM evaluation

Statistical methods

Experiment design

HIPAA compliance

Healthcare domain

Tools

Production tooling

LLM systems

Embeddings

Job description

Stealth Startup is seeking a Head of Data Science in the United States to lead and build our first dedicated Data Science function. You will work directly with the CTO and cross-functionally with Engineering and Product to improve AI systems, design evaluation frameworks, and guide production-ready ML improvements.

You will own the data science function, set technical direction, and determine staffing and infrastructure needs as we scale, while remaining hands-on in implementation and model