AI Evaluation Scientist: Metrics & Safety Lead

Steampunk, Inc.; McLean (VA); On-site; USD 105,000 - 145,000; Full time

Job summary

Steampunk seeks an AI Evaluation Scientist to design, implement, and oversee evaluation processes for predictive and generative AI systems. You will develop metrics, test harnesses, and benchmarks aligned with mission requirements and responsible AI.

You will collaborate with AI developers, LLMOps, data scientists, and governance analysts to ensure reliability, safety, and bias mitigation across LLMs, RAG systems, and predictive models.

Qualifications

• Bachelor’s degree and 8+ years of work experience.
• 2+ years evaluating ML models, NLP systems, or generative AI models (LLMs preferred).
• Proficiency in Python and libraries such as PyTorch, Hugging Face, scikit-learn, LangChain.
• Familiarity with evaluation metrics, statistical testing, dataset creation, and experimental design for AI systems.

Responsibilities

• Design and execute evaluation processes for AI models to ensure accuracy, reliability, safety, and alignment with mission requirements.
• Build and maintain automated evaluation scripts, tests, and pipelines to detect performance drift over time.
• Develop benchmark datasets, challenge sets, and scenario-based test cases for mission needs.
• Perform structured error analysis and behavioral audits of LLMs, RAG systems, and predictive models; document findings and recommendations.
• Collaborate with AI developers, LLMOps engineers, and data scientists to support experimentation and model hardening.
• Contribute to human-in-the-loop evaluation workflows and integrate qualitative and quantitative insights in reports.
• Help map evaluation outcomes to responsible AI principles like fairness, transparency, and safety.
• Stay current with emerging evaluation tools, metrics, and research related to LLM assessment and generative AI reliability.

Skills

Python

PyTorch

Hugging Face

scikit-learn

LangChain

OpenTelemetry

LLM evaluation

AI governance

Education

Bachelor's degree + 8+ years experience

Tools

Ragas

DeepEval

Job description

Steampunk seeks an AI Evaluation Scientist to design, implement, and oversee evaluation processes for predictive and generative AI systems. You will develop metrics, test harnesses, and benchmarks aligned with mission requirements and responsible AI.

You will collaborate with AI developers, LLMOps, data scientists, and governance analysts to ensure reliability, safety, and bias mitigation across LLMs, RAG systems, and predictive models.