Senior Psychometrician

Eticas.AI; United States; Remote; USD 207,000 - 276,000; Part time

Job summary

Eticas.AI is seeking independent experts (junior to senior) for remote, hourly engagements focused on AI in Education across Europe and the US. You will design and run rater training, calibration, and reliability analyses while contributing to LLM-vs-human validity work.

Commitment spans November to March, with 12 days across the period and flexible time zones overlapping US afternoons. Responsibilities include specifying comparison frameworks, sign-off on rubrics, and building datasets and

Qualifications

• PhD or equivalent in educational measurement, psychometrics or quantitative ed psych.
• Has designed scoring rubrics for constructed-response or performance tasks and trained raters on them; runs inter-rater reliability, rater severity and generalizability analyses (ICC, kappa, many-facet Rasch or G-theory) in R or Python.
• Has validated automated essay scoring or LLM-as-judge against human ratings, or can show equivalent work.

Responsibilities

• Review written-reasoning rubric v0 and sign off v1 with the calibration design.
• Design and run rater training and calibration (week of Nov 16).
• Run reliability and severity analyses on the written scoring and CPS.
• Specify the LLM-versus-human comparison and its validity argument.
• Review and final validation in March.

Skills

Educational measurement

Rater reliability

Automated scoring validation

R or Python

Education

PhD in Educational Measurement

Tools

R

Python

Job description

Any time zone that overlaps US afternoons — Remote

Individual or Team junior + senior

As Eticas expands its work on AI in Education in Europe and the US, we are looking to engage independent experts with different levels of expertise to make sure we have everyone at the table and different perspectives to assess our work. Engagements are remote, hourly-based and paid, with an initial defined scope.

Our work is focused on understanding how we can measure AI impacts on learning, and whether we have the tools to measure the right things and support the students who need it. We are also working to build standards so we have homogenized, comparable datasets to understand how learning is changing over time, what works and what doesn't, and for whom.

Working with schools, education authorities and edtech developers, we are testing AI scoring of complex student skills (written reasoning, collaborative problem solving, spoken reasoning) against trained teacher raters, and producing metrics and evidence that show when AI in learning can be trusted, and when it can't.

Our focus is on how AI performs on real student work, judged by real teachers, under real classroom conditions. We get as close as possible to the lived experience of learning and measure it, with the same discipline Eticas has applied for over a decade to AI in hiring, health and public services, among others.

This open call is part of our commitment to openness, inclusion and transparency, so that more voices can join this crucial work.

About the role

• Review written-reasoning rubric v0 (Nov 2 to 6) and sign off v1 with the calibration design (Nov 9 to 13);
• Design and run rater training and calibration (week of Nov 16);
• Run reliability and severity analyses on the written scoring (Nov 23 to Dec 7) and on CPS (Jan 11 to 15);
• Specify the LLM-versus-human comparison and its validity argument (Dec 18);
• Review (Jan 18 to 22) and final validation in March.

Who we are looking for (competencies in order of weight)

• PhD or equivalent in educational measurement, psychometrics or quantitative ed psych;
• Has designed scoring rubrics for constructed-response or performance tasks and trained raters on them; runs inter-rater reliability, rater severity and generalizability analyses (ICC, kappa, many-facet Rasch or G-theory) in R or Python;
• Has validated automated essay scoring or LLM-as-judge against human ratings, or can show equivalent work.

Not a fit: program evaluators, survey methodologists without rating work, academics who cannot commit days in November.

12 days between Nov 2 and Mar 20, front-loaded (about 7 days in November and December, 3 in January, 2 in March). Remote; any time zone that overlaps US afternoons.

Compensation

$150-200 per hour depending on seniority and fit, invoiced monthly against hours worked.