Senior AI Evaluation Scientist — Benchmarks & Systems

Oracle; Santa Clara (CA); On-site; USD 115,000 - 235,000; Full time

Benefits offered by this job

Medical, dental, and vision insurance

401(k) Savings with company match

Paid time off and holidays

Parental leave

Job summary

Oracle seeks a Senior Applied Scientist to independently own end-to-end evaluations of foundation models, AI agents, and enterprise AI systems. You will translate customer and product needs into testable hypotheses, design benchmarks, and build reproducible evaluation pipelines that inform leadership decisions.

The role requires strong Python skills, experience with ML frameworks, and the ability to communicate complex results clearly to science, engineering, product, and leadership teams.

Qualifications

• PhD or Master/Bachelor with equivalent experience.
• Experience designing and executing ML experiments with hypotheses, data, metrics and baselines.
• Strong knowledge of ML, DL, NLP, and generative AI methods.
• Hands-on experience evaluating LLMs, foundation models, or ML systems.
• Proficiency in Python and reliable, maintainable code for experiments.

Responsibilities

• Independently own end-to-end evaluations of foundation models, AI agents, and enterprise AI systems.
• Translate customer/product needs into hypotheses, datasets, metrics, baselines and thresholds.
• Design, implement, and maintain benchmarks and evaluation methods.
• Write Python production-grade evaluation code and build reproducible pipelines.
• Analyze model behavior beyond aggregate scores and assess quality, cost, latency, reliability, and safety.

Skills

Python

Machine Learning

Experiment design

Data analysis

Education

PhD in Computer Science, ML, AI, Statistics, Mathematics

Master/Bachelor with equivalent industry experience

Tools

PyTorch

TensorFlow

Hugging Face

NumPy

pandas

Job description

Oracle seeks a Senior Applied Scientist to independently own end-to-end evaluations of foundation models, AI agents, and enterprise AI systems. You will translate customer and product needs into testable hypotheses, design benchmarks, and build reproducible evaluation pipelines that inform leadership decisions.

The role requires strong Python skills, experience with ML frameworks, and the ability to communicate complex results clearly to science, engineering, product, and leadership teams.