Psychometrician

Artemis Connection; Seattle (WA); On-site; USD 120,000 - 160,000; Full time

Job summary

Artemis Connection partners with an education company to hire a detail-oriented Psychometrician to design, validate, and analyze assessments underpinning academic programs.

You will ensure alignment with accrediting body standards, strengthen validity and fairness, and prepare evidence and reports for re-accreditation self-study and site visits. Collaboration with education leadership, product, research, and data teams is essential.

Qualifications

• Master’s or Ph.D. in Psychometrics, Educational Measurement, Quantitative Psychology, Statistics, or a related field.
• Strong foundation in CTT, IRT, and test validity/reliability frameworks.
• Proficiency with statistical software (R, SPSS, SAS, Python) and psychometric-specific tools (BILOG, Winsteps, flexMIRT, jMetrik).
• Experience analyzing large-scale assessment data sets.
• Excellent written communication skills for producing technical reports for both technical and non-technical audiences.

Responsibilities

• Design and develop test items, scales, and assessment instruments aligned with constructs and objectives.
• Conduct item analysis, reliability testing, and validity studies (content, construct, criterion).
• Apply CTT and/or IRT models to evaluate and refine test items.
• Perform equating, standard setting, and scaling for comparability across forms.
• Conduct DIF analysis to address bias across groups.
• Collaborate with subject matter experts to build item banks.
• Analyze assessment data and produce technical reports on psychometric properties.

Skills

Statistical analysis

Technical reporting

Communication skills

Education

Master’s or Ph.D. in Psychometrics or related field

Tools

R

SPSS

SAS

Python

BILOG

Winsteps

flexMIRT

jMetrik

Job description

Artemis Connection is a strategic management consultancy working across the for-profit, public and social sectors. We help clients around the world identify their most pressing strategic issues and we staff teams of strategy consultants to roll up their sleeves and deliver impact. We are passionate about helping innovative and entrepreneurial leaders reach their goals through a customized project-based approach, typically focused around:

• Bespoke Innovation, Sales, and Marketing Strategy
• Purpose-driven Transformation
• Embedded Strategy and Operations roles

Our founder is Christy Johnson, an entrepreneur, educator, and former McKinsey Engagement Manager. Our advisors include HR officers, executive coaches, academics, entrepreneurs, and neuroscientists. Our team is made up of seasoned consultants, trained at organizations such as McKinsey & Company, Boston Consulting Group (BCG), Bain, Big 4 Strategy, and elite educational institutions. To learn more about Artemis Connection, visit http://artemisconnection.com/.

About the Role

Our client, an education company, is seeking a detail-oriented Psychometrician to lead the design, validation, and analysis of the assessments underpinning its academic programs. This engagement is directly tied to the organization’s upcoming re-accreditation review: the Psychometrician will ensure that assessment practices, evidence, and documentation meet accrediting body standards, while also strengthening the validity, reliability, and fairness of assessments for all test-takers.

Key Responsibilities

• Design and develop test items, scales, and assessment instruments aligned with defined constructs and learning/behavioral objectives
• Conduct item analysis, reliability testing, and validity studies (content, construct, criterion)
• Apply Classical Test Theory (CTT) and/or Item Response Theory (IRT) models to evaluate and refine test items
• Perform equating, standard setting, and scaling to ensure comparability across test forms and administrations
• Conduct differential item functioning (DIF) analysis to identify and address potential bias across demographic groups
• Partner with subject matter experts to build and review item banks
• Analyze assessment data and produce technical reports summarizing psychometric properties, score interpretations, and recommendations
• Ensure assessments comply with relevant professional standards (e.g., APA/AERA/NCME Standards for Educational and Psychological Testing) and legal/regulatory requirements
• Compile and organize psychometric evidence, technical documentation, and validity/reliability reports needed to support the re-accreditation self-study and site visit
• Serve as the assessment subject-matter expert in preparing responses to accreditor inquiries and addressing any prior findings or areas of concern
• Support the design of pilot studies and field tests for new or revised instruments
• Collaborate with academic leadership, product, research, and data teams to translate psychometric findings into actionable improvements
• Stay current on advances in measurement theory, testing technology, and accreditation requirements in the education sector

Qualifications

Required:

• Master’s or Ph.D. in Psychometrics, Educational Measurement, Quantitative Psychology, Statistics, or a related field
• Strong foundation in CTT, IRT, and test validity/reliability frameworks
• Proficiency with statistical software (e.g., R, SPSS, SAS, Python) and psychometric-specific tools (e.g., BILOG, Winsteps, flexMIRT, jMetrik)
• Experience analyzing large-scale assessment data sets
• Excellent written communication skills for producing technical reports for both technical and non-technical audiences

Preferred:

• Experience with computerized adaptive testing (CAT) or automated test assembly
• Familiarity with fairness and bias detection methodologies (e.g., DIF, disparate impact analysis)
• Background in credentialing, certification, educational, or workforce assessment contexts
• Prior experience supporting an accreditation self-study, site visit, or accreditor response process
• Experience presenting findings to stakeholders, academic leadership, or accreditation committees