Proficiency Assessment Architect

Speak; Northern (KY); Hybrid; USD 140,000 - 180,000; Full time

Job summary

In this role you will drive validity plans, collaborate with Product and ML engineers, translate qualitative goals into technical rubrics, and own the quality bar for the scoring system in a fast-growing EdTech environment.

Qualifications

• Assessment/Psychometric Design: 4+ years designing rubrics, blueprints, and item specs for a real, shipped language assessment product.
• Language Proficiency Domain Expertise: Deep familiarity with CEFR/ACTFL descriptors and what separates learning from speaking quality.
• Fairness Across Learner Populations: Identify biases and ensure fair scoring across diverse L1 backgrounds.
• Translates Qualitative → Technical: Convert constructs into concrete, testable specs for ML pipelines.
• Quantitative Rigor: Understand reliability, validity, and basic IRT concepts to judge scoring systems.
• Ownership of Quality Bar: Sign-off on content validity and ensure rigorous quality checks before shipping.
• AI Fluency & Judgment: Apply AI tools judiciously in rubric development and scoring workflows.
• Comfort with Ambiguity: Operate in a 0-to-1 startup environment and turn fuzzy goals into concrete plans.

Responsibilities

• Own Assessment Design — define what Speak measures across Curriculum Mastery, Proficiency, and Placement tests, aligning to CEFR or equivalents; separate constructs like fluency, pronunciation, grammar, and task achievement.

Skills

Assessment design

Language proficiency domain

Fairness across learners

Translate qualitative to technical

Quantitative rigor

Quality ownership

AI familiarity

Ambiguity tolerance

Job description

In this role you will drive validity plans, collaborate with Product and ML engineers, translate qualitative goals into technical rubrics, and own the quality bar for the scoring system in a fast-growing EdTech environment.