Assistant Research Scientist (PREP0004176)

Johns Hopkins University; Baltimore (MD); On-site; USD 55,000 - 75,000; Full time

Benefits offered by this job

Comprehensive health and retirement benefits

Career development support

Job summary

Johns Hopkins University is seeking a PREP Research Associate to support research on human and LLM annotations for AI risk assessment. The successful candidate will collaborate across functions to develop an annotation framework aimed at assessing AI risks.

The ideal candidate should have a background in Computer Science or Data Science, possess a Bachelor’s or Graduate degree, and show strong interest in data annotation. This position offers rewarding opportunities within a supportive environment.

Qualifications

• Bachelor's or Graduate degree in a related field is required.
• Experience or coursework in data annotation and AI risk assessment.
• Ability to collaborate with diverse stakeholders.

Responsibilities

• Familiarize with literature on data annotation and LLMs.
• Contribute to developing an annotation framework.
• Collaborate with interdisciplinary stakeholders for project success.

Skills

Background in Computer Science, Data Science, or a related field

Strong interest in data annotation and AI risks

Familiarity with scientific reading and technical writing

Education

Bachelor's degree or Graduate degree

Job description

PREP Research Associate

This position is part of the National Institute of Standards and Technology (NIST) Professional Research Experience Program (PREP). The PREP program involves staff from a range of backgrounds conducting scientific research across various fields. This role supports a collaboration in which we investigate the reliability of human and large language model annotations for AI risk assessment.

Research Title

Reliability of Human and LLM Annotations for AI Risk Assessment

Key Responsibilities

• Gain familiarity with existing literature on data annotation and the use of LLMs as judges.
• Understand NIST’s role and ongoing efforts in assessing the validity and reliability of AI‑related risks in data annotation.
• Contribute to developing an annotation framework for AI risk assessment.
• Collaborate effectively with cross‑functional and interdisciplinary stakeholders to ensure successful project outcomes.

Deliverables

• Contributions to a NIST report supporting ongoing AI evaluation efforts focusing on the design of an inter‑annotator agreement to assess the reliability of human and LLM annotations.

Qualifications

• Background in Computer Science, Data Science, or a related field.
• Education level: Bachelor’s degree or Graduate degree.
• Strong interest in data annotation and AI risks.
• Familiarity with scientific reading and technical writing.
• U.S. Citizen Preferred.

Salary Range

The referenced salary range represents the minimum and maximum salaries for this position and is based on Johns Hopkins University’s good faith belief at the time of posting. The actual compensation offered to the selected candidate may vary and will ultimately depend on multiple factors, including the successful candidate’s geographic location, skills, work experience, internal equity, market conditions, education/training and other factors, as reasonably determined by the University.

Total Rewards

Johns Hopkins offers a total rewards package that supports our employees’ health, life, career and retirement. More information can be found here: https://hr.jhu.edu/benefits-worklife/

Equal Opportunity Employer

The Johns Hopkins University is committed to equal opportunity for its faculty, staff, and students. The university does not discriminate on the basis of sex, gender, marital status, pregnancy, race, color, ethnicity, national origin, age, disability, religion, sexual orientation, gender identity or expression, veteran status or other legally protected characteristics. The university is committed to providing qualified individuals access to all academic and employment programs, benefits and activities on the basis of demonstrated ability, performance and merit without regard to personal factors or demographic characteristics that are irrelevant to the program involved.

EEO Statement

For more information: https://www.eeoc.gov/employees-job-applicants