Senior Psychometrician

National Board of Medical Examiners; Philadelphia (Philadelphia County); Remote; USD 139,000 - 173,000; Full time

Benefits offered by this job

Remote friendly workplace

Healthcare plans

401(k) match

Tuition reimbursement

Commuter benefits

Job summary

National Board of Medical Examiners is seeking an experienced Senior Psychometrician to advance operational assessment programs and research. The role emphasizes AI and emerging analytics within psychometrics, with collaboration across data analysis, test development, IT, and program management.

The position supports NBME’s remote work model with occasional onsite attendance near Philadelphia and travel as needed.

Qualifications

• PhD in Educational Measurement, Psychological Measurement, Statistics, or related field.
• Minimum 5 years in psychometrics or related measurement role; preference for operational testing experience.
• Experience translating results into clear visual and actionable insights.

Responsibilities

• Lead applied research to enhance examination design and psychometric methods.
• Plan and evaluate psychometric analyses for operational programs (scoring, equating, scaling).
• Develop responsible AI/ML approaches for assessment processes.
• Advance educational measurement through publications and presentations.
• Assess current psychometric processes to identify AI/automation opportunities.
• Create data visualizations for stakeholder reports and interactive tools.

Skills

IRT/CTT

Measurement theory

Data visualization

NLP / data analytics

SAS / Python / R

Communication

Collaboration

AI/ML in assessment

Education

Doctoral degree in Educational Measurement or related field

Tools

SAS

Python

R

Job description

POSITION OVERVIEW

NBME's Psychometrics and Data Analysis (PADA) unit is seeking an experienced Senior Psychometrician to join its team. We are particularly interested in candidates with experience or a demonstrated interest in using artificial intelligence and emerging analytical approaches to support operational assessment processes.

The Senior Psychometrician will apply advanced measurement expertise to support NBME's operational assessment programs, research initiatives, and continued advancement of psychometric practices. This position will contribute across six primary areas:

• Lead and contribute to applied research focused on enhancing examination design, psychometric methodologies, and analytical procedures.
• Plan, conduct, and evaluate psychometric analyses supporting operational examination programs, including scoring, equating, scaling, and score reporting activities.
• Apply psychometric and measurement expertise to develop and evaluate responsible uses of artificial intelligence, machine learning, and emerging analytical methodologies in operational assessment processes.
• Advance the educational measurement field through professional presentations, publications, and research aligned with both individual and organizational research priorities.
• Assess current psychometric processes and methodologies to identify opportunities, including appropriate uses of AI and automation, to improve the accuracy, efficiency, and effectiveness of examination processing.
• Develop and evaluate data visualizations and graphical displays that clearly communicate assessment information through stakeholder reports, examinee feedback, and interactive data tools.

The Senior Psychometrician will work collaboratively across NBME with professionals in areas such as data analysis, test development, information technology, and program management. The position also involves external communication and collaboration with clients and subject matter experts representing a broad range of medical and health care disciplines.

This role has been designated as primarily remote, with occasional onsite attendance required for team collaboration, meetings, or other business needs. We are prioritizing applicants within a 50-mile radius of our Philadelphia office, while also considering qualified candidates in other states where NBME is authorized to employ staff.

Please note that onsite interviews and onboarding at our Philadelphia office may be required for this role. If applicable, advance notice will be provided to support planning.

At NBME, we continue to innovate and improve how we fulfill the evolving needs of the health care community. This commitment starts and ends with the people at NBME. By recruiting and empowering talented individuals from various disciplines and backgrounds, which includes professionals with diverse life experiences, abilities, and perspectives, NBME can take a well-informed, robust approach to advancing medical education and assessment for years to come.

QUALIFICATIONS

Skills and Abilities

• Strong foundation in psychometric theory and its application to operational assessment, including expertise in item response theory (IRT), classical test theory (CTT), statistical methods, test development practices, and research design.
• Advanced knowledge of measurement methodologies such as equating, scaling, generalizability theory, Bayesian methods, and/or standard setting.
• Demonstrated ability to translate complex assessment results into clear and meaningful graphical displays that support appropriate interpretation, decision-making, and action.
• Knowledge of or experience with natural language processing, process-data analytics, formative assessment methodologies, and related emerging approaches.
• Strong technical and computational skills, including experience using statistical and psychometric software and programming tools such as calibration software, SAS, Python, and/or R.
• Excellent written and verbal communication skills, with the ability to explain technical measurement concepts to audiences with varying levels of psychometric expertise.
• Strong analytical and problem-solving skills, including the ability to identify data, methodological, and operational issues and develop appropriate solutions.
• Excellent interpersonal and collaboration skills, with demonstrated success working with colleagues and stakeholders from diverse professional, technical, and educational backgrounds.

Experience

• A minimum of 5 years of professional experience in a Psychometrician, Measurement Scientist, or comparable applied measurement role is required for consideration. Experience supporting operational testing or assessment programs is strongly preferred.

Education

• A doctoral degree in Educational Measurement, Psychological Measurement, Statistics, or a closely related quantitative field is required.

ABOUT NBME

NBME is a not-for-profit organization that specializes in the creation of assessments and learning tools for physicians and health professionals. Our mission is to advance assessment of these professionals to achieve optimal care for all, supporting the development of a highly effective, diverse and compassionate health care workforce.

Founded in 1915, NBME develops and manages the United States Medical Licensing Examination with the Federation of State Medical Boards. We offer a comprehensive portfolio of assessment products for every stage of the medical school journey and provide assessment services to various health profession organizations.

We are dedicated to advancing innovative assessment approaches through research, collaboration and contributions to the medical education and assessment communities. In 2024, we expanded our assessment capabilities to include simulation through the acquisition of MedVR Education.

Learn more on NBME's website.

NBME'S COMMUNITY COLLABORATIONS AND CONTRIBUTIONS

NBME believes that a key path to meeting our mission is collaborating with and supporting our communities. Our Community Collaborations and Contributions programs provide resources and recognition to medical educators and researchers advancing the field of assessment throughout their careers. In addition, we invest in fee assistance, scholarships and pathway programs to aid learners and help support increased representation in health care. Headquartered in Philadelphia, NBME also gives to local organizations that advance health equity and access and positively impact social determinants of health.

The compensation we are offering ranges from $138,688 to $173,360. Please note that the offered rate for this position typically aligns with the minimum to midrange of this position, but it can vary based on the successful candidate's qualifications and experience, department budget, and an internal equity review.

The NBME offers competitive salaries, excellent benefits, and a rewarding work environment. Excellent Benefits include: Healthcare, Dental, Prescription, and Vision plans; 401(k) w/match, Tuition Reimbursement Plan, Commuter Benefit: Public Transit or Parking options. Remote Friendly Workplace.

• Healthcare, Dental, Prescription, and Vision plans
• 401(k) w/match
• Tuition Reimbursement Plan
• Commuter Benefit: Public Transit or Parking options
• Remote Friendly Workplace

NBME is an Equal Opportunity Employer. We will consider all qualified applicants for employment without regard to race; color; sex; gender identity; sexual orientation; religious practices and observances; national origin; pregnancy, childbirth, or related medical conditions; status as a protected veteran or spouse/family member of a protected veteran; or disability.