Proficiency Assessment Architect

Speak; United States; On-site; USD 120,000 - 180,000; Full time

Job summary

Speak is building a cutting-edge, AI-powered language learning platform and seeks an Assessment Design Lead to shape how we measure speaking proficiency. You will own constructs, rubrics, and validity plans, ensuring fairness and reliability of scores across diverse learner populations while collaborating closely with product and ML teams.

You will translate abstract goals into concrete scoring pipelines, define mastery thresholds, and help integrate AI tools without compromising quality.

Qualifications

• 4+ years designing rubrics, blueprints, and item specs for a real, shipped language assessment product.
• Understanding reliability and validity in plain language and how to catch a test measuring the wrong construct.
• Experience ensuring fairness across diverse learner populations in assessment content.
• Ability to translate qualitative goals into concrete scoring constructs for ML pipelines.
• Comfort interpreting reliability statistics and basic IRT concepts for scoring systems.
• Willingness to sign off on content validity and own the quality bar.
• Experience using AI thoughtfully in workflow to support assessment design.
• Ability to operate in a 0-to-1 environment and drive momentum.

Responsibilities

• Own Assessment Design across three test types, focusing on Proficiency Test initially, with expansion to others over time.
• Define constructs, rubrics, and blueprints for reliable scoring aligned to CEFR or equivalent standards.
• Sign off on validity, establish mastery thresholds, and guide rubric guidelines for human labels and ML models.
• Design validity evidence plans including benchmarking studies against external proficiency measures.
• Collaborate with Product and ML teams on automated scoring, calibration, and feedback generation.

Skills

Psychometric design

CEFR familiarity

Fairness in testing

Qualitative to technical

Quantitative rigor

Quality ownership

AI fluency

Ambiguity tolerance

Job description

Speak is building a cutting-edge, AI-powered language learning platform and seeks an Assessment Design Lead to shape how we measure speaking proficiency. You will own constructs, rubrics, and validity plans, ensuring fairness and reliability of scores across diverse learner populations while collaborating closely with product and ML teams.

You will translate abstract goals into concrete scoring pipelines, define mastery thresholds, and help integrate AI tools without compromising quality.