Senior Psychometrics & AI Education Specialist (Remote)

Eticas.AI; United States; Remote; USD 207,000 - 276,000; Part time

Job summary

Eticas.AI is seeking independent experts (junior to senior) for remote, hourly engagements focused on AI in Education across Europe and the US. You will design and run rater training, calibration, and reliability analyses while contributing to LLM-vs-human validity work.

Commitment spans November to March, with 12 days across the period and flexible time zones overlapping US afternoons. Responsibilities include specifying comparison frameworks, sign-off on rubrics, and building datasets and

Qualifications

• PhD or equivalent in educational measurement, psychometrics or quantitative ed psych.
• Has designed scoring rubrics for constructed-response or performance tasks and trained raters on them; runs inter-rater reliability, rater severity and generalizability analyses (ICC, kappa, many-facet Rasch or G-theory) in R or Python.
• Has validated automated essay scoring or LLM-as-judge against human ratings, or can show equivalent work.

Responsibilities

• Review written-reasoning rubric v0 and sign off v1 with the calibration design.
• Design and run rater training and calibration (week of Nov 16).
• Run reliability and severity analyses on the written scoring and CPS.
• Specify the LLM-versus-human comparison and its validity argument.
• Review and final validation in March.

Skills

Educational measurement

Rater reliability

Automated scoring validation

R or Python

Education

PhD in Educational Measurement

Tools

R

Python

Job description

Eticas.AI is seeking independent experts (junior to senior) for remote, hourly engagements focused on AI in Education across Europe and the US. You will design and run rater training, calibration, and reliability analyses while contributing to LLM-vs-human validity work.

Commitment spans November to March, with 12 days across the period and flexible time zones overlapping US afternoons. Responsibilities include specifying comparison frameworks, sign-off on rubrics, and building datasets and