Chief Assessment Officer

The American Board of Anesthesiology; Raleigh (NC); Hybrid; USD 180,000 - 240,000; Full time

Job summary

The American Board of Anesthesiology (ABA) seeks a Chief Assessment Officer to lead the science, strategy, and innovation behind its assessment portfolio. This executive will guide 33 staff across Assessment Development, Psychometrics and Research, Credentialing, and Assessment Delivery.

The role focuses on digital-first assessment, AI integration, and high-quality patient care, reporting to the Executive Director with a hybrid schedule in Raleigh.

Qualifications

• A doctoral degree in Educational Measurement, Psychometrics, Educational Psychology, or a closely related field.
• At least 10 years of progressively responsible leadership in assessment, credentialing, or a related field.
• Executive leadership experience overseeing high-stakes assessment programs with psychometric expertise.
• Experience leading multidisciplinary teams and significant organizational transformation.
• Experience partnering with technology teams to advance assessment platforms and AI.
• Strong strategic, financial management, and communication capabilities.

Responsibilities

• Oversee the complete assessment lifecycle from design to reporting.
• Lead credentialing and continuing certification initiatives integrated with assessment strategy.
• Direct validation and research efforts to evaluate program effectiveness.
• Collaborate with technology teams on platform requirements and governance for AI in assessment.

Skills

Psychometrics

Executive leadership

Assessment design

Statistical methodologies

Strategic thinking

Communication

Education

Doctoral degree in Educational Measurement or related discipline

Job description

Hybrid schedule with a minimum of two days per week on-site

Reports to the Executive Director

Help shape the future of physician assessment and certification.

The American Board of Anesthesiology (ABA) seeks a Chief Assessment Officer (CAO) to lead the science, strategy, and innovation behind its assessment, credentialing, and certification programs. This is an opportunity for an accomplished psychometric expert and executive leader to advance how physician knowledge and performance are evaluated by combining scientific rigor, emerging technology, and a commitment to high-quality patient care.

Since 1938, the ABA has served the anesthesiology profession through certification. Today, the organization is building on that foundation with an innovative approach to assessment and continuous professional development. Its mission is to advance the highest standards of anesthesiology practice and help physicians provide the best possible care for every patient, every day.

The Opportunity

As a member of the Executive Leadership Team, the CAO will help shape enterprise strategy while overseeing the development, validation, delivery, and continuous improvement of the ABA’s assessment portfolio. The role leads approximately 33 staff through three direct reports across Assessment Development, Psychometrics and Research, Credentialing, and Assessment Delivery.

The incoming leader will help guide the ABA toward a digital-first assessment ecosystem by exploring artificial intelligence, adaptive approaches, simulation, and assessment methods that extend beyond traditional multiple-choice examinations. Success requires recognized credibility in measurement science and the executive judgment to translate innovation into valid, reliable, fair, and accessible certification programs.

This leader must think beyond departmental priorities, collaborate across the organization, and build trust with physician leaders, governing board members, and an experienced multidisciplinary team.

What You Will Lead

• Enterprise strategy and assessment innovation. Partner with the Executive Leadership Team and Board of Directors to shape long-range priorities, evolve certification models, and identify approaches that strengthen the ABA’s mission and physician experience.
• Assessment and psychometric excellence. Oversee the complete assessment lifecycle, including examination design, development, administration, psychometric analysis, scoring, and reporting.
• Credentialing and continuing certification. Lead the credentialing and continuing certification enterprise, ensuring that eligibility, certification status, pathway requirements, and diplomate services are integrated with assessment strategy and program operations.
• Research leadership. Direct validation and research efforts that evaluate program effectiveness, inform organizational decisions, and advance the ABA’s contributions to assessment and medical education.
• Responsible technology advancement. Partner with technology leaders to establish platform requirements and investment priorities, including the validation, governance, and ethical use of artificial intelligence in assessment development and scoring.
• Operational and financial performance. Oversee budgets, vendors, resource planning, quality measures, and process improvements that support secure, scalable, and efficient assessment delivery.
• Talent and organizational leadership. Develop high-performing leaders and teams, guide change thoughtfully, and foster a culture of collaboration, inclusion, accountability, and continuous learning.

The Leader We Seek

The successful candidate will bring deep expertise in psychometrics and high-stakes assessment, along with the ability to lead at the enterprise level. This person can explain complex scientific concepts clearly, build productive partnerships across disciplines, and distinguish promising innovation from approaches that require further evidence. The ideal candidate will be equally comfortable advising a governing board, developing people, and translating strategy into operational results.

Qualified candidates will bring:

• A doctoral degree, such as a Ph.D., Ed.D., Psy.D., or equivalent, in Educational Measurement, Psychometrics, Educational Psychology, Industrial and Organizational Psychology, Statistics, Measurement and Evaluation, or a closely related discipline.
• At least 10 years of progressively responsible leadership experience in assessment, credentialing, certification, licensure, educational measurement, or a related field.
• Executive leadership experience overseeing complex, high-stakes assessment programs, with demonstrated expertise in psychometric theory, assessment design, research, and statistical methodologies.
• Experience leading multidisciplinary teams and significant organizational transformation.
• Experience partnering with technology teams to advance assessment platforms and digital products, with knowledge of artificial intelligence, advanced analytics, and emerging assessment technologies.
• Strong strategic, financial management, communication, and organizational development capabilities.

Experience working with physician volunteers, governing boards, or professional associations is strongly preferred.

Working at the ABA

The ABA is a fast-paced organization that continuously innovates to meet the evolving needs of anesthesiologists and the profession. The organization offers a welcoming and inclusive environment that values professional growth, flexibility, collaboration, and continuous learning. The ABA embraces change as an opportunity to solve meaningful challenges and improve how it serves physicians.

This position requires an on-site presence in Raleigh, North Carolina, at least two days per week. Occasional travel of up to 10%, including overnight stays, is also required.