Senior AI Evaluation Scientist

Oracle; Nashville (TN); On-site; USD 115,000 - 235,000; Full time

Benefits offered by this job

Medical, dental, vision insurance

401(k) with company match

Paid time off

Job summary

Oracle’s OCI AI Evaluation team seeks a Senior Applied Scientist to own complex evaluation work from problem definition to final recommendation. You will translate questions into measurable hypotheses, design benchmarks, and build evaluation pipelines.

You will work with researchers, engineers, product teams, and leadership to deliver defensible results. You will write robust Python code, validate data and metrics, and examine factors beyond aggregate scores such as robustness, data provenance,

Qualifications

• PhD in Computer Science, ML, AI, Statistics or related quantitative field; or Master/Bachelor with equivalent industry experience.
• Experience designing and executing ML experiments, defining hypotheses, datasets, metrics, baselines, and interpreting results.
• Strong knowledge of modern ML, deep learning, NLP, and generative AI methods.
• Hands-on experience evaluating LLMs, foundation models, AI agents, or related systems.
• Proficiency in Python and production-oriented experiment code.
• Experience with ML frameworks and data-science libraries (PyTorch, TensorFlow, HuggingFace, NumPy, pandas).
• Ability to analyze complex datasets, perform statistical and error analyses, and identify data-quality issues.
• Experience translating technical findings into clear recommendations for diverse stakeholders.
• Ability to work independently while collaborating with cross-functional teams.
• Strong written and verbal communication skills.

Responsibilities

• Own end-to-end evaluations of foundation models and AI systems, from design to analysis and stakeholder review.
• Translate customer and business needs into testable hypotheses, datasets, and metrics.
• Design and maintain benchmarks and evaluation methods across ML capabilities.
• Write production-oriented evaluation code and build reproducible pipelines and checks.
• Evaluate model behavior across quality, cost, latency, reliability, safety, and domain fit.
• Conduct statistical and failure-mode analyses to explain results and differences.
• Develop automated evaluators, including LLM-as-a-judge methods, against human judgments.
• Design human-evaluation workflows with rubrics and quality controls.

Skills

PhD in CS/ML

Experiment design

Statistical analysis

Technical writing

Independent work

Strong communication

Education

PhD in Computer Science/ML

Master/Bachelor with equivalent experience

Tools

PyTorch

TensorFlow

HuggingFace

NumPy

pandas

Job description

Oracle’s OCI AI Evaluation team seeks a Senior Applied Scientist to own complex evaluation work from problem definition to final recommendation. You will translate questions into measurable hypotheses, design benchmarks, and build evaluation pipelines.

You will work with researchers, engineers, product teams, and leadership to deliver defensible results. You will write robust Python code, validate data and metrics, and examine factors beyond aggregate scores such as robustness, data provenance,