AI Evaluation Scientist: Trusted AI Metrics & Tests

Steampunk, Inc.; McLean (VA); On-site; USD 105,000 - 145,000; Full time

Job summary

Steampunk seeks an AI Evaluation Scientist to design and execute evaluation processes ensuring AI systems are accurate, reliable, safe, and aligned with mission requirements.

You will collaborate with AI Developers, LLMOps Engineers, and Data Scientists to develop evaluation metrics, build test harnesses, analyze model behavior, and support responsible deployment across the AI lifecycle.

Qualifications

• Bachelor’s degree and 5+ years of experience.
• Master’s degree and 3+ years of experience.
• 2+ years evaluating ML models, NLP systems, or generative AI models (LLMs preferred).
• Proficiency in Python and relevant libraries (PyTorch, Hugging Face, scikit-learn, LangChain).
• Familiarity with evaluation metrics, statistical testing, dataset creation, and experimental design.

Responsibilities

• Design evaluation processes to ensure accuracy, reliability, safety, and alignment with mission requirements.
• Develop benchmarks, test harnesses, and pipelines to monitor performance drift.
• Perform structured error analysis and audits of LLMs, RAG, and predictive models.

Skills

Python

PyTorch

Hugging Face

scikit-learn

LangChain

Education

Bachelor's degree in CS/Stats/ML/etc.

Master's degree in CS/Stats/ML/etc.

Tools

OpenTelemetry

Ragas/DeepEval

Job description

Steampunk seeks an AI Evaluation Scientist to design and execute evaluation processes ensuring AI systems are accurate, reliable, safe, and aligned with mission requirements.

You will collaborate with AI Developers, LLMOps Engineers, and Data Scientists to develop evaluation metrics, build test harnesses, analyze model behavior, and support responsible deployment across the AI lifecycle.