AI Evaluation Scientist: Trust & Safety Leader

Steampunk; McLean (VA); Hybrid; USD 105,000 - 145,000; Full time

Job summary

Steampunk seeks an AI Evaluation Scientist to design and execute evaluation processes for predictive and generative AI systems, ensuring accuracy, reliability, safety, and alignment with mission requirements. You will work with engineers, data scientists, governance analysts, and product teams to develop metrics, build test harnesses, and analyze model behavior.

Responsibilities include implementing evaluation frameworks, automating tests, and documenting results for technical and non-technical

Qualifications

• Bachelor’s or Master’s degree in CS, Stats, ML, Cognitive Science, HCI or Data Science.
• 2+ years evaluating ML models, NLP systems, or generative AI models (LLMs preferred).
• Proficiency in Python and libraries such as PyTorch, Hugging Face, scikit-learn, LangChain.
• Experience with evaluation frameworks like Ragas or DeepEval, and OpenTelemetry tools.
• Ability to interpret model weaknesses and provide actionable improvements.
• Strong written and verbal communication; comfortable with agile environments.

Responsibilities

• Implement evaluation frameworks for AI models, including accuracy, robustness, bias, safety metrics.
• Build automated evaluation scripts, tests, and pipelines to detect performance drift.
• Develop benchmark datasets and scenario-based tests aligned with mission needs.
• Perform structured error analysis and behavioral audits of LLMs, RAG, and predictive models.
• Collaborate with AI Developers, LLMOps Engineers, and Data Scientists for experimentation and hardening.
• Design human-in-the-loop evaluation workflows and document results for audiences.
• Map evaluation outcomes to responsible AI principles and governance requirements.
• Stay current with emerging evaluation tools, metrics, and research for LLMs.
• Document evaluation processes for technical and non-technical audiences.

Skills

Public trust eligibility

Analytical skills

Communication

Agile / iterative

Responsible AI

OWASP LLM Top 10

Education

Degree in CS / Statistics / ML / Cognitive Science / Data Science

Tools

Python

PyTorch

Hugging Face

scikit-learn

LangChain

OpenTelemetry

Ragas

DeepEval

Job description

Steampunk seeks an AI Evaluation Scientist to design and execute evaluation processes for predictive and generative AI systems, ensuring accuracy, reliability, safety, and alignment with mission requirements. You will work with engineers, data scientists, governance analysts, and product teams to develop metrics, build test harnesses, and analyze model behavior.

Responsibilities include implementing evaluation frameworks, automating tests, and documenting results for technical and non-technical