Senior Foundation-Model Evaluation Scientist

Oracle; Seattle (WA); On-site; USD 115,000 - 235,000; Full time

Benefits offered by this job

401(k) matching

Employee Stock Purchase Plan

Paid time off

Paid holidays

Sick leave

Job summary

Oracle is seeking a Senior Applied Scientist to independently own end-to-end evaluations of foundation models, AI agents, and enterprise AI systems. You will translate customer needs into testable hypotheses, design benchmarks, and build reproducible evaluation pipelines.

Seek to analyze data quality, latency, cost, and reliability while collaborating with science, engineering, product, and leadership teams. Strong Python and ML framework experience required.

Qualifications

• PhD in Computer Science, Machine Learning, Artificial Intelligence, Statistics, Mathematics, or a related quantitative field; or a Master's or Bachelor's degree with equivalent relevant industry experience.
• Experience designing and executing machine learning experiments, including defining hypotheses, selecting datasets and metrics, establishing baselines, and interpreting results.
• Strong knowledge of modern machine learning, deep learning, natural language processing, and generative AI methods.
• Hands-on experience evaluating large language models, foundation models, AI agents, or other machine learning systems.
• Proficiency in Python and experience writing reliable, maintainable code for experiments, data processing, or machine learning systems.

Responsibilities

• Independently own end-to-end evaluations of foundation models, AI agents, and enterprise AI systems; from design through analysis and stakeholder review.
• Translate customer, product, and business needs into testable hypotheses, evaluation criteria, datasets, metrics, baselines, and acceptance thresholds.
• Design, implement, and maintain benchmarks and evaluation methods for reasoning, coding, agentic workflows, RAG, NL2SQL, multimodal systems, multilingual performance, and responsible AI.
• Write high-quality Python and production-oriented evaluation code; build reproducible pipelines, test suites, automated checks, and integrations with shared evaluation platforms.
• Evaluate model and system behavior across quality, cost, latency, reliability, safety, robustness, and domain fit.

Skills

Python

Experiment design

Data analysis

Communication

Education

PhD in computer science / ML / AI / statistics or related field

Master's or Bachelor's with equivalent industry experience

Tools

PyTorch

TensorFlow

Hugging Face

NumPy

pandas

Job description

Oracle is seeking a Senior Applied Scientist to independently own end-to-end evaluations of foundation models, AI agents, and enterprise AI systems. You will translate customer needs into testable hypotheses, design benchmarks, and build reproducible evaluation pipelines.

Seek to analyze data quality, latency, cost, and reliability while collaborating with science, engineering, product, and leadership teams. Strong Python and ML framework experience required.