Interim Vice-President, Psychometrics and Data Analytics

The Pharmacy Examining Board of Canada; Canada; Remote; CAD 200,000 - 260,000; Full time

Job summary

The Pharmacy Examining Board of Canada (PEBC) is seeking an Interim Vice-President, Psychometrics and Data Analytics to redefine psychometric and data strategies and provide senior leadership across governance, analytics, reporting and data strategy. You will ensure validity, reliability and defensibility of national certification examinations while treating data as a strategic asset.

Reporting to the CEO and serving on the Senior Management Team, you will advance psychometric practices, lead

Qualifications

• PhD required, ideally in psychometrics or educational measurement, or in a closely related quantitative discipline such as statistics.
• Significant progressive senior leadership and management experience, including leading technical teams or functions.
• Substantial experience as a psychometrician in educational or psychological testing (typically eight or more years), including high-stakes assessment.
• Strong expertise in psychometric methodologies, including standard setting, equating and scaling.
• Experience leading or overseeing enterprise data strategy, data governance, data analytics and business intelligence/reporting.
• Proven ability to design, conduct, interpret and present psychometric analyses and studies in high-stakes assessment contexts.
• Strong data analysis and scientific-computing skills, and familiarity with business-intelligence and data-visualization tools.
• Experience leveraging data science, machine learning or emerging AI tools to support measurement and analytics.
• Assets: exposure to predictive/prescriptive analytics and AI/ML model governance; professional certification and licensure experience; proficiency in French.

Responsibilities

• Oversee the psychometric quality of all assessments, validity, reliability, fairness and defensibility, through test design and administration support, standard setting, equating, scaling, scoring and results reporting.
• Ensure consistent application of psychometric methodologies across assessments, in line with professional standards and best practices.
• Direct statistical and forensic analyses of examination outcomes, candidate performance and testing incidents (including remote proctoring) to support fair, evidence-based decisions.
• Lead the initial development and implementation of an enterprise data governance framework, data quality, integrity, security, privacy, retention, definitions and stewardship, alongside metadata management, a data catalogue, and master and reference data management.
• Govern analytical and AI/machine-learning models used in or affecting assessments, including model validation, bias and fairness testing, performance monitoring, drift detection and documentation.
• Own the reporting, dashboard and data-visualization environment, including self-serve business intelligence for senior leadership, the Board and stakeholders, underpinned by a governed single source of truth and semantic layer.
• Champion a data strategy that treats assessment and operational data as a strategic asset, and advance analytics from descriptive and diagnostic reporting toward predictive and prescriptive capability.
• Steward the multi-year data strategy and analytics roadmap aligned to the strategic plan, in collaboration with technology leadership, which owns the underlying data platform and engineering.
• Chair the data governance operating model, with named data owners and stewards in each function, reporting through the Senior Management Team to the CEO.
• Advise on psychometric and standard-setting decisions with high-stakes or policy-level impact, and on statistical and forensic findings related to testing irregularities.
• Partner closely with executive peers in Assessments, Technology and Operations, and present psychometric results, validity evidence and recommendations to the Board's examinations committee.
• Lead all HR matters for the Psychometrics and Data Analytics department, including recruitment, training, retention and performance management, along with annual business priorities, projects and budget.

Skills

Psychometrics

Leadership

Data analytics

Business intelligence

Statistical analysis

French (asset)

Education

PhD in psychometrics or educational measurement or related quantitative discipline

Tools

Business intelligence tools

Statistical computing

AI/ML tools

Job description

The Pharmacy Examining Board of Canada (PEBC), a national not-for-profit certification and assessment body for the pharmacy profession, is seeking an Interim Vice-President, Psychometrics and Data Analytics.

This is a highly strategic executive mandate. PEBC is looking for a strong, proven leader who can redefine its psychometric and data strategies and provide senior leadership across psychometrics, enterprise data strategy, analytics, governance and business intelligence.

Reporting to the Chief Executive Officer and serving on the Senior Management Team, the Interim Vice-President holds overall accountability for psychometrics and related data analysis, and leads PEBC’s enterprise data strategy, data analytics and business intelligence, data governance, and reporting. The role ensures the validity, reliability, fairness and defensibility of high-stakes national certification examinations, and positions data as a strategic asset informing decision-making across the organization.

Combining strategic leadership with deep technical authority, the Interim Vice-President advances psychometric and analytic practice through established and emerging methodologies, research and evidence-informed decision-making.

Key Responsibilities

• Oversee the psychometric quality of all assessments, validity, reliability, fairness and defensibility, through test design and administration support, standard setting, equating, scaling, scoring and results reporting.
• Ensure consistent application of psychometric methodologies across assessments, in line with professional standards and best practices.
• Direct statistical and forensic analyses of examination outcomes, candidate performance and testing incidents (including remote proctoring) to support fair, evidence-based decisions.
• Lead the initial development and implementation of an enterprise data governance framework, data quality, integrity, security, privacy, retention, definitions and stewardship, alongside metadata management, a data catalogue, and master and reference data management.
• Govern analytical and AI/machine-learning models used in or affecting assessments, including model validation, bias and fairness testing, performance monitoring, drift detection and documentation.
• Own the reporting, dashboard and data-visualization environment, including self-serve business intelligence for senior leadership, the Board and stakeholders, underpinned by a governed single source of truth and semantic layer.
• Champion a data strategy that treats assessment and operational data as a strategic asset, and advance analytics from descriptive and diagnostic reporting toward predictive and prescriptive capability.
• Steward the multi-year data strategy and analytics roadmap aligned to the strategic plan, in collaboration with technology leadership, which owns the underlying data platform and engineering.
• Chair the data governance operating model, with named data owners and stewards in each function, reporting through the Senior Management Team to the CEO.
• Advise on psychometric and standard-setting decisions with high-stakes or policy-level impact, and on statistical and forensic findings related to testing irregularities.
• Partner closely with executive peers in Assessments, Technology and Operations, and present psychometric results, validity evidence and recommendations to the Board's examinations committee.
• Lead all HR matters for the Psychometrics and Data Analytics department, including recruitment, training, retention and performance management, along with annual business priorities, projects and budget.

Qualifications

• PhD required, ideally in psychometrics or educational measurement, or in a closely related quantitative discipline such as statistics.
• Significant progressive senior leadership and management experience, including leading technical teams or functions.
• Substantial experience as a psychometrician in educational or psychological testing (typically eight or more years), including high-stakes assessment.
• Strong expertise in psychometric methodologies, including standard setting, equating and scaling.
• Experience leading or overseeing enterprise data strategy, data governance, data analytics and business intelligence/reporting.
• Proven ability to design, conduct, interpret and present psychometric analyses and studies in high-stakes assessment contexts.
• Strong data analysis and scientific-computing skills, and familiarity with business-intelligence and data-visualization tools.
• Experience leveraging data science, machine learning or emerging AI tools to support measurement and analytics.
• Assets: exposure to predictive/prescriptive analytics and AI/ML model governance; professional certification and licensure experience; proficiency in French.

The Ideal Candidate

You are a strong, seasoned leader with a track record of setting direction, someone who has redefined strategy for a technical function, not just managed one. You bring high professional standards and a strong commitment to assessment quality, fairness and defensibility, with sound analytical judgment on complex, high-impact psychometric and data issues. You balance strategic leadership with technical rigour, hands-on where needed and treat data integrity, security and privacy as paramount. You are a credible, confident and collaborative communicator, able to translate complex psychometric and data concepts for non-technical audiences, inspire and develop a technical team, and operate independently in a high-stakes assessment environment.

Working Conditions

This is a fully remote position open to candidates located in Canada or the United States. Occasional travel may be required.

PEBC is an equal opportunity employer committed to an inclusive, barrier-free recruitment process. Accommodations are available on request at all stages of the selection process.

This role is a replacement, and AI will not be used during the recruitment process.