Tech Lead Data Scientist, AI Evaluation & Monitoring

Geisinger; Pennsylvania; Hybrid; USD 120,000 - 150,000; Full time

Benefits offered by this job

Healthcare benefits from day one

Vision and dental coverage

Benefits for domestic partners

Job summary

Geisinger is seeking a Tech Lead Data Scientist, responsible for evaluating and optimizing AI systems in production. You will lead technical direction for AI evaluation across various programs, guiding teams in conducting validation studies and monitoring plans.

The ideal candidate has extensive experience in data science, ML engineering, and excels in technical leadership without formal authority. Preferred qualifications include an advanced degree in a quantitative field and experience in healthcare or regulated industries.

Qualifications

• 6+ years in data science, statistics, ML engineering, or applied quantitative research with senior technical experience.
• Hands-on experience designing and running model evaluation studies in real production settings.
• Fluency in Python and SQL; comfortable with modern ML tooling and cloud-native data environments.

Responsibilities

• Own the technical evaluation methodology across the enterprise.
• Guide program teams in designing validation studies and monitoring plans.
• Develop and maintain evaluation toolkits for AI systems.

Skills

Data science

Machine learning engineering

Statistical analysis

Experimental design

Python programming

SQL proficiency

Fairness evaluation for ML systems

Technical leadership

Education

Bachelor’s Degree in a related field

MS or PhD in a quantitative field

Job description

Job Summary

The Tech Lead Data Scientist, AI Evaluation & Monitoring is the principal technical expert responsible for evaluating, monitoring, and optimizing AI systems in production at Geisinger. This hands‑on technical leadership role sets technical direction for AI evaluation across a growing portfolio, leads a team of data analysts, and partners with AI program teams to improve how AI is validated, monitored, and improved over time.

Responsibilities

• Own the technical evaluation methodology across the enterprise, including pre‑production validation, production monitoring, and ongoing optimization.
• Guide program teams in designing validation studies, equity audits, monitoring plans, and escalation playbooks for their AI systems.
• Translate program‑specific failure modes into concrete, measurable metrics and instrument production monitoring.
• Develop and maintain evaluation toolkits such as LLM‑as‑Judge frameworks, golden sets, simulation harnesses, experimental study designs, drift detection, and subgroup fairness analysis.
• Create reusable evaluation playbooks and templates that accelerate new program onboarding.
• Provide technical direction, design review, and mentorship for a team of data analysts.
• Coach teams to produce rigorous evaluation deliverables for governance review.

What You Will Not Own

• People management or formal performance evaluations for the analyst team.
• Program‑level product strategy or go/no‑go decisions.
• Final clinical validation judgment for AI safety.
• Software infrastructure behind evaluation tooling (handled by the AI Platform team).

Required Skills & Qualifications

• 6+ years in data science, statistics, ML engineering, or applied quantitative research, with senior technical experience on cross‑functional projects.
• Strong foundation in experimental design and causal inference, with judgment on method selection.
• Hands‑on experience designing and running model evaluation studies in real production settings.
• Experience evaluating LLM or generative AI systems, or comparable complex ML systems with imperfect ground truth.
• Demonstrated ability to translate ambiguous failure modes into concrete evaluation designs and monitoring metrics.
• Fluency in Python and SQL; comfortable with modern ML tooling and cloud‑native data environments.
• Experience with fairness and equity evaluation for ML systems.
• Track record of providing technical leadership and mentorship without formal people‑management authority.
• Clear written communication skills for producing evaluation memos and specifications relied upon by non‑technical decision makers.
• Clinical, healthcare, or regulated‑industry experience strongly preferred.

Preferred Qualifications

• MS or PhD in a quantitative field, or equivalent experience.
• Education: Bachelor’s Degree in a related field (required).

Benefits

• Healthcare benefits for full‑time and part‑time employees from day one, including vision, dental, and domestic partners.

Equal Opportunity Employer

Geisinger is an affirmative action, equal‑opportunity employer and all qualified applicants will receive consideration for employment regardless of race, color, religion, sex, sexual orientation, gender identity, national origin, disability or status as a protected veteran.