AI Evaluation Scientist — LLMs & Systems

Pacific Northwest National Laboratory; Richland, Seattle (WA, WA); On-site; USD 114,000 - 182,000; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

Pension plan

Relocation

Job summary

Pacific Northwest National Laboratory (PNNL) invites applications for a Data Scientist in the AI and Data Analytics Division, onsite in Richland, WA or Seattle, WA.

You will design and execute evaluation studies for AI/ML, prepare datasets and scenarios, develop evaluation environments and harnesses, run controlled experiments, and analyze model behavior, including aspects of explainability, robustness, and uncertainty.

Qualifications

• BS/BA with 2+ years data science experience, or MS/MA, or PhD.
• Experience developing or evaluating LLMs, VLMs, or AI agents.
• Experience using Python and ML frameworks.

Responsibilities

• Implement test and evaluation activities for AI/ML with transformer models.
• Prepare evaluation datasets, prompts, mission scenarios, datasets and baselines.
• Develop evaluation environments and harnesses for models and tools.
• Execute controlled experiments at model, component, workflow, and system levels.
• Analyze model behavior for weaknesses, robustness, and data sensitivity.
• Document methods and results in reproducible code and technical reports.

Skills

Python

ML frameworks

Education

BS/BA with 2+ years data science OR MS/MA OR PhD

Job description

Pacific Northwest National Laboratory (PNNL) invites applications for a Data Scientist in the AI and Data Analytics Division, onsite in Richland, WA or Seattle, WA.

You will design and execute evaluation studies for AI/ML, prepare datasets and scenarios, develop evaluation environments and harnesses, run controlled experiments, and analyze model behavior, including aspects of explainability, robustness, and uncertainty.