Medical Assessment Test Architect

ISHE; East Lansing, Northern (MI, KY); Hybrid; USD 80,000 - 120,000; Full time

Job summary

ABEM is seeking a Test Developer to draft exams, review quality, and maintain integrity of medical assessments. You will guide programs through all development stages, coordinating SMEs, editors, and psychometricians to ensure alignment with blueprints and standards.

Responsibilities include managing item banks, form development, committee logistics, staff training, and ongoing policy improvements. Travel up to 15% may be required as part of the role.

Qualifications

• Four-year degree and 3–5 years of relevant work experience in testing or a related field.
• At least 2 years of program or project management experience; 5 years preferred.
• Understanding of question writing and examination construction principles.
• Understanding of measurement principles.
• Ability to collect and analyze data to inform decisions.
• Ability to learn medical terminology.
• Proficiency with Microsoft 365 apps (Excel, PowerPoint, SharePoint, Word); advanced Excel preferred.
• Strong oral and written communication skills.
• Strong problem-solving and critical-thinking skills with sound judgement.
• Maintain professionalism and confidentiality about sensitive matters.
• Collaborate with teams for continuous quality improvement.
• Experience with web-based item banking applications (e.g., ExamDeveloper), test delivery platforms, and project management tools (Trello, Smartsheet, HubSpot).
• Ability to travel up to 15% of the time.

Responsibilities

• Guide examination programs by developing, updating, and coordinating schedules with SMEs and staff editors.
• Maintain item banks across programs using standard processes and gap analyses.
• Develop examination forms aligned with blueprints and selection criteria; coordinate with psychometricians.
• Guide SMEs through development stages: form assembly, review, publication, validation, and post-exam reviews.
• Organize exam development committee meetings; set agendas and timelines.
• Prepare materials for committee, task force, and Board review.
• Assist onboarding and training of new SMEs; develop training materials with editors.
• Foster relationships with committee members and staff to achieve outcomes.
• Ensure quality by sharing test development best practices.
• Contribute to standard policies and procedures for test development.
• Consult with psychometricians for scoring tasks and related projects.
• Develop and maintain test development reports for stakeholders.
• Provide technical support for testing applications and software updates.
• Plan and run annual Item Writers Workshop; provide on-site support and SME management.
• Staff case development meetings and support ongoing activities.
• Maintain documentation of item/case development and processes; identify improvements.
• Collaborate in problem-solving and addressing issues as they arise.
• Represent ABEM in presentations to stakeholders and professional audiences.
• Staff certification examinations and related meetings.

Skills

Program/project management

Question writing & exam construction

Measurement principles

Data analysis

Medical terminology

Communication skills

Problem solving

Professionalism & confidentiality

Team collaboration

ExamDeveloper

Trello

Smartsheet

HubSpot

Travel readiness

Microsoft 365

Education

Four-year degree

Tools

Microsoft 365

ExamDeveloper

Trello

Smartsheet

HubSpot

Job description

ABEM is seeking a Test Developer to draft exams, review quality, and maintain integrity of medical assessments. You will guide programs through all development stages, coordinating SMEs, editors, and psychometricians to ensure alignment with blueprints and standards.

Responsibilities include managing item banks, form development, committee logistics, staff training, and ongoing policy improvements. Travel up to 15% may be required as part of the role.