Tech Lead Data Scientist, AI Evaluation & Monitoring

Remotedxb; Dubai; On-site; AED 300,000 - 520,000; Full time

Benefits offered by this job

Healthcare benefits including vision &

Domestic partner benefits

Job summary

Geisinger is seeking an experienced data science leader to set the technical direction for AI evaluation across a large portfolio of production systems. You will design and operate the evaluation toolkit, including LLM-as-Judge frameworks, golden sets, and simulation harnesses.

You will provide hands-on guidance to program teams on validation studies, equity audits, and monitoring plans, translate ambiguous failure modes into concrete, measurable production metrics, and offer technical

Qualifications

• 6+ years of experience in data science, statistics, ML engineering, or applied quantitative research.
• Strong foundation in experimental design and causal inference.
• Hands-on experience designing and running model evaluation studies in production settings.
• Experience evaluating LLM, generative AI, or complex ML systems.
• Proficiency in Python and SQL and comfort with cloud-native data environments.
• Experience with fairness and equity evaluation for ML systems.
• Ability to provide technical leadership without formal people-management authority.

Responsibilities

• Set the technical direction for AI evaluation across a large portfolio of production systems
• Design and operate the evaluation toolkit, including LLM-as-Judge frameworks, golden sets, and simulation harnesses
• Provide hands-on guidance to program teams on validation studies, equity audits, and monitoring plans
• Translate ambiguous failure modes into concrete, measurable production monitoring metrics
• Provide technical leadership, design review, and mentorship to a team of data analysts

Skills

Data science

ML engineering

Experimental design

Causal inference

Python

SQL

Model evaluation

Leadership

Fairness evaluation

Education

MS or PhD in a quantitative field

Job description

Responsibilities

• Set the technical direction for AI evaluation across a large portfolio of production systems
• Design and operate the evaluation toolkit, including LLM-as-Judge frameworks, golden sets, and simulation harnesses
• Provide hands‑on guidance to program teams on validation studies, equity audits, and monitoring plans
• Translate ambiguous failure modes into concrete, measurable production monitoring metrics
• Provide technical leadership, design review, and mentorship to a team of data analysts

Requirements

• 6+ years of experience in data science, statistics, ML engineering, or applied quantitative research
• Strong foundation in experimental design and causal inference
• Hands‑on experience designing and running model evaluation studies in production settings
• Experience evaluating LLM, generative AI, or complex ML systems
• Proficiency in Python and SQL and comfort with cloud‑native data environments
• Experience with fairness and equity evaluation for ML systems
• Ability to provide technical leadership without formal people‑management authority

Preferred Qualifications

• MS or PhD in a quantitative field
• Experience in healthcare, clinical, or regulated industries

Benefits

• Healthcare benefits including vision and dental from day one
• Domestic partner benefits

About the Company

Geisinger is a healthcare system serving more than 1 million people, comprising nine hospital campuses, a health plan, two research centers, and the Geisinger Commonwealth School of Medicine.