Remote Assessment Design Lead – Language Proficiency & AI

Ladders

United States

Remote

USD 108,000 - 130,000

Full time

21 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Remote work
Competitive compensation
Impactful language assessments

Job summary

Ladders is seeking an Assessment Design Lead to shape the strategy behind language learning assessments. The role defines what is measured, how often, and how those measures connect to learner outcomes, translating goals into clear scoreable constructs for item writers, data science, and ML partners.

You will ensure validity, fairness, and reliability across diverse learner populations while guiding responsible AI use in assessment.

Qualifications

  • 4+ years of assessment or psychometric design experience in language testing.
  • Solid knowledge of language proficiency frameworks CEFR, ACTFL, IELTS, TOEFL.
  • Ability to identify bias and ensure fair scoring across diverse populations.
  • Experience translating qualitative constructs into actionable metrics for ML.
  • Statistical analysis for reliability and validity, including inter-rater reliability.
  • Experience using AI tools in assessment design.

Responsibilities

  • Define assessment types and their frequency, focusing on proficiency tests.
  • Translate abstract goals into scoreable constructs and rubrics for writers and ML engineers.
  • Ensure content validity across assessments and identify measurement errors before launch.
  • Design a validity evidence plan to benchmark assessments against external measures.
  • Collaborate with Product and ML teams to align constructs with automated scoring systems.

Skills

Psychometric design
Language assessments
Bias detection
AI in assessment
Reliability and validity
Translation to ML metrics

Job description

Ladders is seeking an Assessment Design Lead to shape the strategy behind language learning assessments. The role defines what is measured, how often, and how those measures connect to learner outcomes, translating goals into clear scoreable constructs for item writers, data science, and ML partners.

You will ensure validity, fairness, and reliability across diverse learner populations while guiding responsible AI use in assessment.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Assessment Design Lead
Assessment Design Lead

Ladders • United States

Remote
USD 108,000 - 130,000
Remote work
Competitive compensation
Impactful language assessments
Proficiency Assessment Architect
Proficiency Assessment Architect

Speak • United States

On-site
USD 120,000 - 180,000
Language Assessment Scientist for High-Impact Tests
Language Assessment Scientist for High-Impact Tests

Duolingo • Pittsburgh

On-site
USD 111,000 - 150,000
Lead AI Systems Engineer for Educational Assessment
Lead AI Systems Engineer for Educational Assessment

Pearson • Carson City (NV)

On-site
USD 140,000 - 160,000
Senior AI Systems Engineer, Education & Assessment
Senior AI Systems Engineer, Education & Assessment

Pearson • Lansing (MI)

On-site
USD 140,000 - 160,000
Lead Specialist, AI Scientist
Lead Specialist, AI Scientist

Pearson • Lansing (MI)

On-site
USD 140,000 - 160,000
Lead Specialist, AI Scientist
Lead Specialist, AI Scientist

Pearson • Carson City (NV)

On-site
USD 140,000 - 160,000
Lead Specialist, AI Scientist
Lead Specialist, AI Scientist

Pearson • Springfield (IL)

On-site
USD 140,000 - 160,000
Lead Specialist, AI Scientist
Lead Specialist, AI Scientist

Pearson • Oklahoma City (OK)

On-site
USD 140,000 - 160,000
Lead Specialist, AI Scientist
Lead Specialist, AI Scientist

Pearson • United States

Remote
USD 140,000 - 160,000