Tech Lead, LLM Evaluation & Production Monitoring

Remotedxb; Dubai; On-site; AED 300,000 - 520,000; Full time

Benefits offered by this job

Healthcare benefits including vision &

Domestic partner benefits

Job summary

Geisinger is seeking an experienced data science leader to set the technical direction for AI evaluation across a large portfolio of production systems. You will design and operate the evaluation toolkit, including LLM-as-Judge frameworks, golden sets, and simulation harnesses.

You will provide hands-on guidance to program teams on validation studies, equity audits, and monitoring plans, translate ambiguous failure modes into concrete, measurable production metrics, and offer technical

Qualifications

• 6+ years of experience in data science, statistics, ML engineering, or applied quantitative research.
• Strong foundation in experimental design and causal inference.
• Hands-on experience designing and running model evaluation studies in production settings.
• Experience evaluating LLM, generative AI, or complex ML systems.
• Proficiency in Python and SQL and comfort with cloud-native data environments.
• Experience with fairness and equity evaluation for ML systems.
• Ability to provide technical leadership without formal people-management authority.

Responsibilities

• Set the technical direction for AI evaluation across a large portfolio of production systems
• Design and operate the evaluation toolkit, including LLM-as-Judge frameworks, golden sets, and simulation harnesses
• Provide hands-on guidance to program teams on validation studies, equity audits, and monitoring plans
• Translate ambiguous failure modes into concrete, measurable production monitoring metrics
• Provide technical leadership, design review, and mentorship to a team of data analysts

Skills

Data science

ML engineering

Experimental design

Causal inference

Python

SQL

Model evaluation

Leadership

Fairness evaluation

Education

MS or PhD in a quantitative field

Job description

Geisinger is seeking an experienced data science leader to set the technical direction for AI evaluation across a large portfolio of production systems. You will design and operate the evaluation toolkit, including LLM-as-Judge frameworks, golden sets, and simulation harnesses.

You will provide hands-on guidance to program teams on validation studies, equity audits, and monitoring plans, translate ambiguous failure modes into concrete, measurable production metrics, and offer technical