Senior Director, Assessment (opens in new tab)

Test Publishers; United States; Remote; USD 140,000 - 210,000; Full time

Job summary

PA Education Association seeks a Senior Director, Assessment to lead a high-performing team in shaping the organization's assessment strategy and products. The role guides psychometric and measurement activities, collaborates with executives, and directs vendor partnerships to ensure valid, reliable, and defensible assessments.

The position emphasizes innovation, strategic leadership, and mission-driven impact within health professions education, with responsibilities spanning development,

Qualifications

• Lead the strategy, planning, implementation, evaluation, and continuous improvement of current and future assessment products and services.
• Monitor developments in educational measurement, psychometrics, assessment technology, AI, and health professions education to identify opportunities to improve assessment design, delivery, reporting, and product development.
• Collaborate with organizational leaders on assessment initiatives, business planning, and product development opportunities.

Responsibilities

• Oversee the full assessment lifecycle, including blueprint development and evaluation, item banking, test construction, administration and delivery, scoring, scaling, equating, standard setting, reporting, quality assurance, and technical documentation.
• Ensure assessments are valid, reliable, fair, accessible, secure, and defensible, and comply with testing standards, policies, and governance requirements.
• Lead psychometric analyses and measurement research, including item and test performance monitoring, scoring methodology, standard setting, applied research, corrective action, and continuous evaluation.
• Guide the development and use of data tools, analytical solutions, and measurement technologies, and communicate findings to technical and nontechnical audiences.

Skills

Psychometrics

Educational measurement

Leadership

Strategic planning

Job description

The PA Education Association (PAEA) is seeking a Senior Director, Assessment to lead, develop, and grow a high-performing assessment team while guiding the organization's assessment strategy. The ideal candidate is an experienced people leader with a proven record of building, mentoring, and expanding teams that deliver excellence, collaboration, and continuous improvement. This leader also brings a strong foundation in psychometrics and educational measurement and a commitment to innovation and mission-driven impact.

The Senior Director, Assessment provides strategic and operational leadership for PAEA's assessment portfolio, translating organizational goals into assessment strategy and operational plans and ensuring that high-quality assessment programs and services are developed, delivered, evaluated, and continuously improved. Working closely with the Chief Data Officer and Principal Psychometrician, this position oversees assessment operations, psychometric and measurement activities, exam development, innovation initiatives, and external partnerships and vendors, ensuring that PAEA assessments remain valid, reliable, fair, defensible, and aligned with organizational strategy. The Senior Director also supports the development of future assessment products and services and represents PAEA within the broader assessment, measurement, and health professions education communities.

Key Requirements

Assessment Strategy and Innovation

• Lead the strategy, planning, implementation, evaluation, and continuous improvement of current and future assessment products and services, aligning short- and long-term priorities with organizational goals and member needs.
• Monitor developments in educational measurement, psychometrics, assessment technology, artificial intelligence, and health professions education to identify opportunities to improve assessment design, delivery, reporting, and product development.
• Collaborate with organizational leaders on assessment initiatives, business planning, and product development opportunities.

Assessment Program and Measurement Oversight

• Oversee the full assessment lifecycle, including blueprint development and evaluation, item banking, test construction, administration and delivery, scoring, scaling, equating, standard setting, reporting, quality assurance, and technical documentation.
• Ensure assessments are valid, reliable, fair, accessible, secure, and legally defensible, and comply with testing standards, industry best practices, organizational policies, and governance requirements. Establish and maintain related assessment policies and procedures.
• Lead psychometric analyses and measurement research, including item and test performance monitoring, scoring methodology, standard setting, applied research, corrective action, and continuous evaluation, to support evidence-based decision-making.
• Guide the development and use of data tools, analytical solutions, and measurement technologies, and communicate technical findings, score interpretation guidance, and assessment information to technical and nontechnical audiences.

Team Leadership and Development

• Supervise, coach, and mentor Assessment staff; manage performance; and foster alignment across psychometric, operational, and strategic functions.
• Support staff development and participation in professional organizations, conferences, publications, and research, while strengthening organizational capacity through cross-training, knowledge sharing, and succession planning.

Operations and Partnerships

• Oversee exam development activities, production schedules, budgets, resource allocation, volunteer engagement, and vendor performance.
• Coordinate the work of assessment staff, subject matter experts, vendors, and external partners, and cultivate relationships that advance PAEA’s assessment strategy.
• Implement process improvements that enhance quality, efficiency, scalability, sustainability, and the stakeholder experience, and support assessment-related meetings, summits, workshops, and professional development activities.

External and Professional Leadership

• Represent PAEA through presentations, publications, workshops, consultation, and professional service within the testing, psychometrics, educational measurement, credentialing, and health professions education communities.
• Strengthen members’ assessment literacy through educational programming, consultation, score interpretation resources, and the dissemination of best practices.

Security and Compliance

• Protect the confidentiality, integrity, and security of assessment materials, data, and processes, and maintain related policies, procedures, and risk controls.
• Coordinate incident response, risk mitigation, and compliance activities with organizational leaders, legal counsel, and other relevant stakeholders.

Other Duties

• The duties and responsibilities listed above are representative of the nature and level of work assigned and are not necessarily all-inclusive. Duties, responsibilities, and activities may change at any time with or without notice.