Chief Assessment Officer (The ABA)

Nonprofit HR; Raleigh (NC); Hybrid; USD 180,000 - 240,000; Full time

Job summary

The American Board of Anesthesiology (ABA) seeks a Chief Assessment Officer to lead the science, strategy, and innovation behind its assessment, credentialing, and certification programs in Raleigh, NC. This executive leader will oversee 33 staff and drive a digital-first, AI-enabled assessment ecosystem while maintaining rigorous measurement standards.

Ideal candidates hold a doctoral degree in a relevant field, 10+ years in high-stakes assessment leadership, and a track record of

Qualifications

• Doctoral degree in Educational Measurement, Psychometrics, Educational Psychology, or related field.
• 10+ years of progressively responsible leadership in assessment, credentialing, or related area.
• Executive leadership experience with high-stakes assessment programs and data-driven decisions.

Responsibilities

• Lead enterprise strategy and innovation across certification models and assessment programs.
• Oversee full assessment lifecycle: design, development, administration, psychometrics, scoring, reporting.
• Direct credentialing, continuing certification, and diplomate services integration with operations.
• Advance validation, research, and measurement science to inform decisions.
• Guide technology strategy, AI governance, and digital product development for assessments.
• Manage budgets, vendor relationships, and scalable delivery of programs.
• Develop leaders, foster collaboration, and promote a culture of inclusion and continuous learning.

Skills

Executive leadership

Psychometrics

Assessment design

Research methods

Strategic planning

Financial management

Stakeholder communication

Team leadership

AI in assessment

Analytics

Education

Doctoral degree in Educational Measurement / Psychometrics / related field

Tools

AI in assessment

Advanced analytics

Emerging assessment technologies

Job description

Chief Assessment Officer

Raleigh, North Carolina
Hybrid schedule with a minimum of two days per week on-site
Reports to the Executive Director

Help shape the future of physician assessment and certification.

The American Board of Anesthesiology (ABA) seeks a Chief Assessment Officer (CAO) to lead the science, strategy, and innovation behind its assessment, credentialing, and certification programs. This is an opportunity for an accomplished psychometric expert and executive leader to advance how physician knowledge and performance are evaluated by combining scientific rigor, emerging technology, and a commitment to high-quality patient care.

Since 1938, the ABA has served the anesthesiology profession through certification. Today, the organization is building on that foundation with an innovative approach to assessment and continuous professional development. Its mission is to advance the highest standards of anesthesiology practice and help physicians provide the best possible care for every patient, every day.

The Opportunity

As a member of the Executive Leadership Team, the CAO will help shape enterprise strategy while overseeing the development, validation, delivery, and continuous improvement of the ABA’s assessment portfolio. The role leads approximately 33 staff through three direct reports across Assessment Development, Psychometrics and Research, Credentialing, and Assessment Delivery.

The incoming leader will help guide the ABA toward a digital-first assessment ecosystem by exploring artificial intelligence, adaptive approaches, simulation, and assessment methods that extend beyond traditional multiple-choice examinations. Success requires recognized credibility in measurement science and the executive judgment to translate innovation into valid, reliable, fair, and accessible certification programs.

This leader must think beyond departmental priorities, collaborate across the organization, and build trust with physician leaders, governing board members, and an experienced multidisciplinary team.

What You Will Lead

• Enterprise strategy and assessment innovation. Partner with the Executive Leadership Team and Board of Directors to shape long-range priorities, evolve certification models, and identify approaches that strengthen the ABA’s mission and physician experience.
• Assessment and psychometric excellence. Oversee the complete assessment lifecycle, including examination design, development, administration, psychometric analysis, scoring, and reporting.
• Credentialing and continuing certification. Lead the credentialing and continuing certification enterprise, ensuring that eligibility, certification status, pathway requirements, and diplomate services are integrated with assessment strategy and program operations.
• Research leadership. Direct validation and research efforts that evaluate program effectiveness, inform organizational decisions, and advance the ABA’s contributions to assessment and medical education.
• Responsible technology advancement. Partner with technology leaders to establish platform requirements and investment priorities, including the validation, governance, and ethical use of artificial intelligence in assessment development and scoring.
• Operational and financial performance. Oversee budgets, vendors, resource planning, quality measures, and process improvements that support secure, scalable, and efficient assessment delivery.
• Talent and organizational leadership. Develop high-performing leaders and teams, guide change thoughtfully, and foster a culture of collaboration, inclusion, accountability, and continuous learning.

The Leader We Seek

The successful candidate will bring deep expertise in psychometrics and high-stakes assessment, along with the ability to lead at the enterprise level. This person can explain complex scientific concepts clearly, build productive partnerships across disciplines, and distinguish promising innovation from approaches that require further evidence. The ideal candidate will be equally comfortable advising a governing board, developing people, and translating strategy into operational results.

Qualified candidates will bring:

• A doctoral degree, such as a Ph.D., Ed.D., Psy.D., or equivalent, in Educational Measurement, Psychometrics, Educational Psychology, Industrial and Organizational Psychology, Statistics, Measurement and Evaluation, or a closely related discipline.
• At least 10 years of progressively responsible leadership experience in assessment, credentialing, certification, licensure, educational measurement, or a related field.
• Executive leadership experience overseeing complex, high-stakes assessment programs, with demonstrated expertise in psychometric theory, assessment design, research, and statistical methodologies.
• Experience leading multidisciplinary teams and significant organizational transformation.
• Experience partnering with technology teams to advance assessment platforms and digital products, with knowledge of artificial intelligence, advanced analytics, and emerging assessment technologies.
• Strong strategic, financial management, communication, and organizational development capabilities.

Experience working with physician volunteers, governing boards, or professional associations is strongly preferred.

Working at the ABA

The ABA is a fast-paced organization that continuously innovates to meet the evolving needs of anesthesiologists and the profession. The organization offers a welcoming and inclusive environment that values professional growth, flexibility, collaboration, and continuous learning. The ABA embraces change as an opportunity to solve meaningful challenges and improve how it serves physicians.

This position requires an on-site presence in Raleigh, North Carolina, at least two days per week. Occasional travel of up to 10%, including overnight stays, is also required.

Our Commitment to Diversity, Equity, Inclusion & Belonging

Nonprofit HR is committed to fostering and maintaining a work environment where diversity, equity, and inclusion (DEI) are fully integrated into everything we do for the benefit of our employees and the clients that we serve. To fully realize our goal, we prioritize our understanding of the complexities of DEI within our workforce to inform our approach to talent management. We believe that this guides how we do our work, advise our clients to operationalize DEI and position our content and educational opportunities help strengthen the talent management capacity of the social impact sector.

Continue reading our about our commitment at nonprofithr.com/deinow.