Test (Examination) Developer

ISHE; East Lansing, Northern (MI, KY); On-site; USD 80,000 - 120,000; Full time

Job summary

ABEM is seeking a Test Developer to draft exams, review quality, and maintain integrity of medical assessments. You will guide programs through all development stages, coordinating SMEs, editors, and psychometricians to ensure alignment with blueprints and standards.

Responsibilities include managing item banks, form development, committee logistics, staff training, and ongoing policy improvements. Travel up to 15% may be required as part of the role.

Qualifications

• Four-year degree and 3–5 years of relevant work experience in testing or a related field.
• At least 2 years of program or project management experience; 5 years preferred.
• Understanding of question writing and examination construction principles.
• Understanding of measurement principles.
• Ability to collect and analyze data to inform decisions.
• Ability to learn medical terminology.
• Proficiency with Microsoft 365 apps (Excel, PowerPoint, SharePoint, Word); advanced Excel preferred.
• Strong oral and written communication skills.
• Strong problem-solving and critical-thinking skills with sound judgement.
• Maintain professionalism and confidentiality about sensitive matters.
• Collaborate with teams for continuous quality improvement.
• Experience with web-based item banking applications (e.g., ExamDeveloper), test delivery platforms, and project management tools (Trello, Smartsheet, HubSpot).
• Ability to travel up to 15% of the time.

Responsibilities

• Guide examination programs by developing, updating, and coordinating schedules with SMEs and staff editors.
• Maintain item banks across programs using standard processes and gap analyses.
• Develop examination forms aligned with blueprints and selection criteria; coordinate with psychometricians.
• Guide SMEs through development stages: form assembly, review, publication, validation, and post-exam reviews.
• Organize exam development committee meetings; set agendas and timelines.
• Prepare materials for committee, task force, and Board review.
• Assist onboarding and training of new SMEs; develop training materials with editors.
• Foster relationships with committee members and staff to achieve outcomes.
• Ensure quality by sharing test development best practices.
• Contribute to standard policies and procedures for test development.
• Consult with psychometricians for scoring tasks and related projects.
• Develop and maintain test development reports for stakeholders.
• Provide technical support for testing applications and software updates.
• Plan and run annual Item Writers Workshop; provide on-site support and SME management.
• Staff case development meetings and support ongoing activities.
• Maintain documentation of item/case development and processes; identify improvements.
• Collaborate in problem-solving and addressing issues as they arise.
• Represent ABEM in presentations to stakeholders and professional audiences.
• Staff certification examinations and related meetings.

Skills

Program/project management

Question writing & exam construction

Measurement principles

Data analysis

Medical terminology

Communication skills

Problem solving

Professionalism & confidentiality

Team collaboration

ExamDeveloper

Trello

Smartsheet

HubSpot

Travel readiness

Microsoft 365

Education

Four-year degree

Tools

Microsoft 365

ExamDeveloper

Trello

Smartsheet

HubSpot

Job description

POSITION SUMMARY

The Test Developer drafts exams, conducts quality control reviews, and ensures the accuracy and integrity of medical assessments. This role guides assigned examination programs through all stages of test development and production, maintaining high standards for content and compliance.

ESSENTIAL DUTIES AND RESPONSIBILITIES

• Guides multiple examination programs by developing, updating, and coordinating project schedules and assignments for subject matter experts (SMEs) in collaboration with staff editors.
• Maintains item banks for multiple examination programs using standardized processes, conducting gap analyses as needed to ensure complete coverage of all domains in the examination blueprints.
• Develops examination forms for assigned examination programs, ensuring they align with blueprints and selection criteria. Works with psychometricians to establish selection parameters and necessary linkages.
• Guides SMEs through various stages of examination development, including form assembly and review, publication, key validation and post-exam reviews.
• Organizes and guides exam development committee meetings and related logistics, including setting agendas, establishing timelines, and coordinating participant involvement, in collaboration with staff editors.
• Prepares and contributes materials for committee, task force, and Board review.
• Assists with the onboarding and training of new SMEs, including developing and updating training materials in collaboration with staff editors.
• Fosters and maintains strong relationships with committee members and staff to achieve desired outcomes.
• Ensures quality by offering test development expertise and sharing best practices with staff and SMEs.
• Contributes to the creation and upkeep of standard policies and procedures related to test development activities.
• Consults with psychometricians as needed to support scoring tasks (e.g., standard setting, key validation) and other test-related projects (e.g., job task analysis).
• Develops, updates, and maintains test development reports for both internal and external stakeholders.
• Provides technical support for staff and SMEs on testing applications, including testing software updates as needed.
• Participates in the planning and implementation of the annual Item Writers Workshop, including on-site support and management of SMEs.
• Staffs case development meetings and provides ongoing support to case development teams between meetings as assigned.
• Develops and maintains documentation of item/case development and exam development processes while identifying opportunities for process improvements.
• Collaborates in problem-solving by analyzing, documenting, and addressing issues as they arise.
• Represents ABEM in presentations to stakeholders and professional audiences (e.g., ATP, ICE, ABMS Conference).
• Staffs the certification examination administrations and additional meetings as assigned.
• Performs other duties as needed or assigned.

MINIMUM QUALIFICATIONS (KNOWLEDGE, SKILLS, AND ABILITIES)

• Four-year degree and three to five years of relevant work experience in testing or a related field.
• At least two years of program or project management experience is required. Five years is experience is preferred.
• Understanding of question writing and examination construction principles is required. Expertise in this area is preferred.
• Understanding of measurement principles.
• Ability to collect and analyze data to inform decisions.
• Ability to learn medical terminology.
• Proficiency with Microsoft 365 (e.g., Excel, PowerPoint, SharePoint, Word); advanced Excel skills are preferred.
• Strong oral and written communication skills.
• Strong problem-solving and critical-thinking skills, and the ability to exercise sound judgement and make decisions based on accurate and timely analysis.
• Must maintain a high level of professionalism and confidentiality about sensitive matters and all organizational information.
• Ability to collaborate with individuals and teams in pursuit of continuous quality improvement.
• Ability to use web-based item banking applications (e.g., ExamDeveloper), test delivery platforms, and project management software (e.g., Trello, Smartsheet, HubSpot).
• Ability to travel up to 15% percent of the time.

PHYSICAL DEMANDS AND WORK ENVIRONMENT

The physical demands described here are representative of those that must be met by an employee to successfully perform the essential functions of this position. Reasonable accommodation may be made to enable individuals with disabilities to perform these functions.

While performing the duties of this position, the employee is regularly required to talk or listen. The employee frequently is required to use hands or fingers, handle or feel objects, tools, or controls. The employee is occasionally required to stand, walk, sit, and reach with hands and arms. Specific vision abilities required by this position include close vision, distance vision, and the ability to adjust focus.

The noise level in the work environment is typically low to moderate.