GenAI Evaluation Scientist

Johns Hopkins University; Baltimore (MD); On-site; USD 80,000 - 110,000; Full time

Job summary

Johns Hopkins University in Baltimore is seeking a PREP Research Associate to support the NIST GenAI program within the PREP framework. The role focuses on developing AI evaluation infrastructure, organizing data, and contributing to reports aligned with program goals.

Initial work includes installing and deploying software on a GPU-accelerated cluster, evaluating AI modalities, and collaborating to design a robust evaluation framework for generative AI systems.

Qualifications

• Proficiency in languages used for scientific computing and AI development.
• Experience with REST-based applications and networking basics.
• Shell scripting and basic cloud/virtualization knowledge.
• Familiarity with JSON Schema and data formats for evaluation.

Responsibilities

• Install, develop and deploy software for AI/ML systems on GPU clusters.
• Conduct image/audio/video evaluation series for generative AI.
• Design experiments and curate datasets for evaluation.
• Develop baseline algorithms and improve evaluation infrastructure.
• Maintain and enhance NIST GenAI software and tools.
• Manage data formats and translate specs for interoperability.

Skills

Python

Java

JavaScript

Ruby

R

C

C++

Rust

Shell scripting

Education

Master's degree in computer science or related field

Tools

Docker

Kubernetes

AWS

Virtualization

Job description

Johns Hopkins University in Baltimore is seeking a PREP Research Associate to support the NIST GenAI program within the PREP framework. The role focuses on developing AI evaluation infrastructure, organizing data, and contributing to reports aligned with program goals.

Initial work includes installing and deploying software on a GPU-accelerated cluster, evaluating AI modalities, and collaborating to design a robust evaluation framework for generative AI systems.