Senior Applied Scientist - End-to-End AI Evaluations

Oracle; Austin (TX); On-site; USD 115,000 - 234,000; Full time

Benefits offered by this job

Health insurance

401(k) plan

Paid time off

Job summary

Oracle in Austin, TX is seeking a Senior Applied Scientist to lead end-to-end evaluations of foundation models and AI systems. You will translate business questions into rigorous testable hypotheses, design benchmarks, and build scalable evaluation pipelines.

You will work with cross-functional teams to turn research into actionable product decisions and ensure robust, reproducible results. The role emphasizes statistical analysis, code quality, and practical deployments, with collaboration

Qualifications

• PhD in Computer Science, Machine Learning, Artificial Intelligence, Statistics, Mathematics, or related quantitative field; or a Master’s/Bachelor’s with equivalent experience.
• Hands-on experience designing and running ML experiments, defining hypotheses, selecting datasets and metrics, establishing baselines, and interpreting results.
• Strong knowledge of modern ML, deep learning, NLP, and generative AI methods.
• Experience evaluating large language models, foundation models, AI agents, or other ML systems.
• Proficiency in Python and writing reliable, production-ready code for experiments and data processing.

Responsibilities

• Own end-to-end evaluations of foundation models, AI agents, and enterprise AI systems from question design through analysis and stakeholder review.
• Translate business needs into testable hypotheses, evaluation criteria, datasets, metrics, baselines, and thresholds.
• Design and maintain benchmarks and evaluation methods across reasoning, coding, RAG, NL2SQL, multimodal systems, and responsible AI.
• Write production-oriented evaluation code and build reproducible pipelines and test suites.
• Conduct statistical analysis, error analysis, ablations, and failure-mode investigations to explain model behavior.

Skills

Python

Experiment design

Statistical analysis

Communication

Independent work

Education

PhD in CS/ML/related

MS/BS with industry experience

Tools

PyTorch

TensorFlow

Hugging Face

NumPy

pandas

Job description

Oracle in Austin, TX is seeking a Senior Applied Scientist to lead end-to-end evaluations of foundation models and AI systems. You will translate business questions into rigorous testable hypotheses, design benchmarks, and build scalable evaluation pipelines.

You will work with cross-functional teams to turn research into actionable product decisions and ensure robust, reproducible results. The role emphasizes statistical analysis, code quality, and practical deployments, with collaboration