GenAI Research Scientist - AI Evaluation & Systems

Johns Hopkins University; Baltimore (MD); On-site; USD 65,000 - 95,000; Full time

Job summary

Johns Hopkins University is seeking a PREP Research Associate to support the NIST GenAI program in the Information Access Division. The role focuses on developing AI evaluation infrastructure and datasets, with emphasis on deepfake related tasks and measurement science.

You will build and deploy AI/ML software on GPU clusters, design experiments, curate datasets, and contribute to standardized agentic frameworks and evaluation metrics.

Qualifications

• Proficiency in Python, Java, JavaScript, Ruby, R, C, C++, or Rust.

Responsibilities

• Installing, developing and deploying software for executing AI and ML based systems, with a large dataset on a GPU-accelerated cluster
• Conducting image/audio/video evaluation series in advancing generative AI technologies
• Conducting experimental design and dataset curation
• Developing baseline algorithms in generative AI
• Upgrading, maintaining, and continuously developing existing NIST GenAI software and evaluation infrastructure
• Managing data, such as querying and translating data formats/specifications
• Designing and deploying agentic AI evaluation metrics and frameworks to assess agent performance, dynamic planning, tool use accuracy, and error recovery behaviors in complex multi-step workflows as well as multi-agent systems
• Developing standardized baseline agent architectures to serve as control benchmarks within the NIST GenAI framework

Skills

Python

Java

JavaScript

Ruby

R

C

C++

Rust

Networking basics

Shell scripting

Docker/Kubernetes/AWS

CV/Signal processing

Data visualization

Agentic design patterns

JSON Schema / MCP

Agentic eval frameworks

Advanced technical fundamentals

Tools

Docker

Kubernetes

AWS

Job description

Johns Hopkins University is seeking a PREP Research Associate to support the NIST GenAI program in the Information Access Division. The role focuses on developing AI evaluation infrastructure and datasets, with emphasis on deepfake related tasks and measurement science.

You will build and deploy AI/ML software on GPU clusters, design experiments, curate datasets, and contribute to standardized agentic frameworks and evaluation metrics.