Junior Psychometrician

Eticas.AI; United States; Remote; USD 55,000 - 83,000; Part time

Job summary

Eticas.AI seeks independent research experts to work remotely on AI in Education across Europe and the US. The hourly engagement has an initial defined scope and flexible time zones overlapping US afternoons.

You will prepare scoring data, run reliability analyses, produce tables/figures, and keep scripts reproducible. We require a Master’s or PhD student in measurement, educational psychology or statistics who can compute ICC and kappa in R or Python and document what was done.

Qualifications

• Master's or PhD student in measurement, educational psychology or statistics.
• Has computed ICC, kappa and rater-effect models in R or Python.
• Follows a lead's specification without needing it re-explained.
• Documents what they ran.

Responsibilities

• Prepare scoring data from Nov 23.
• Run reliability, agreement and severity analyses for written reasoning (Dec) and CPS (Jan).
• Produce tables and figures.
• Keep analysis scripts reproducible.

Skills

ICC models

Kappa statistics

R

Python

Documentation

Education

Master’s or PhD student in measurement, educational psychology or statistics

Tools

R

Python

Job description

Any time zone that overlaps US afternoons — Remote

Individual or Team junior + senior

As Eticas expands its work on AI in Education in Europe and the US, we are looking to engage independent experts with different levels of expertise to make sure we have everyone at the table and different perspectives to assess our work. Engagements are remote, hourly-based and paid, with an initial defined scope.

Our work is focused on understanding how we can measure AI impacts on learning, and whether we have the tools to measure the right things and support the students who need it. We are also working to build standards so we have homogenized, comparable datasets to understand how learning is changing over time, what works and what doesn't, and for whom.

Working with schools, education authorities and edtech developers, we are testing AI scoring of complex student skills (written reasoning, collaborative problem solving, spoken reasoning) against trained teacher raters, and producing metrics and evidence that show when AI in learning can be trusted, and when it can't.

Our focus is on how AI performs on real student work, judged by real teachers, under real classroom conditions. We get as close as possible to the lived experience of learning and measure it, with the same discipline Eticas has applied for over a decade to AI in hiring, health and public services, among others.

This open call is part of our commitment to openness, inclusion and transparency, so that more voices can join this crucial work.

About the role

Prepare the scoring data from Nov 23;

Run reliability, agreement and severity analyses for written reasoning (Dec) and CPS (Jan);

Produce tables and figures;

Keep analysis scripts reproducible.

Who we are looking for / Competencies

Master's or PhD student in measurement, educational psychology or statistics;

Has computed ICC, kappa and rater-effect models in R or Python;

Follows a lead's specification without needing it re-explained;

Documents what they ran.

Dedication

8 days between Nov 23 and Feb 26, peaking Dec 1 to 11 and Jan 11 to 22. Remote; any time zone that overlaps US afternoons.

Compensation

$40-60 per hour depending on seniority and fit, invoiced monthly against hours worked.