Lead Assessment Developer

Outsmart; Lehi (UT); Hybrid; USD 120,000 - 180,000; Full time

Benefits offered by this job

Competitive salary and equity

Healthcare for dependents

Parental leave

15 PTO days per year

Remote work days

Daily in-office lunch

Working with talented individuals

Job summary

Outsmart, located in Lehi, Utah, is seeking a skilled Assessment Development Manager to oversee the full assessment development lifecycle. You will work closely with various teams to build effective assessments, ensuring adherence to psychometric standards while creating a meaningful impact in higher education. The ideal candidate has a Master's degree (Ph.D. preferred), over three years of experience in managing operational assessments, and strong communication skills. The position offers a salary range of $120K–$180K, along with competitive perks including healthcare, remote work, and PTO days.

Qualifications

• 3+ years managing high-stakes assessment lifecycles end-to-end.
• Proven experience leading standard-setting studies.
• Experience managing external vendors.

Responsibilities

• Manage end-to-end assessment development operations.
• Facilitate standard-setting studies and final reporting.
• Oversee vendor and external partner relationships.

Skills

Leadership in standard-setting studies

Managing operational assessment lifecycles

Communication for non-technical audiences

Knowledge of validity and reliability standards

Education

Master's degree in educational measurement or related field

Ph.D. preferred

Job description

Outsmart is a mission-driven, early-stage startup reimagining higher education for the age of AI by creating an entirely new collegiate experience that expands access and opportunity. Backed by $38M from top venture firms including Khosla Ventures, Forerunner, DST Global Partners, and Lightspeed, the company is led by former Duolingo executives and a founding team that also includes Emmy Award winners. With a strong culture centered around high-impact, detail-oriented team members, Outsmart is already seeing significant traction, generating 1–5 million weekly organic views on social media.

What You’ll Be Doing

You’ll manage the full assessment development lifecycle at Outsmart — from item authoring through field testing, standard setting, scoring, and reporting. You’ll work closely with psychometricians, faculty, product managers, and engineers to build rigorous, scalable assessments. As an AI-forward organization, you’ll also help shape how AI tools are responsibly woven into our development and QA workflows.

Who You Are

• Master's degree (Ph.D. preferred) in educational measurement, assessment, or a related field
• 3+ years managing operational, high-stakes assessment lifecycles end-to-end
• Proven experience leading standard-setting studies and managing diverse SME panels
• Experience managing external vendors for field tests and pilot studies
• Working knowledge of validity, reliability, and fairness standards (e.g., AERA/APA/NCME)
• Familiarity with cognitive complexity frameworks (DOK, Bloom's)
• Strong communicator who can translate psychometric concepts for non-technical audiences
• Bonus: knowledge of CTT, IRT, CDM, DIF, AI-assisted item generation, or accreditation reviews

What You’ll Own

• End-to-end assessment development operations for assigned assessments
• SME network — recruitment, training, calibration, and quality
• Standard-setting studies — method selection, panel facilitation, and final reporting
• Vendor and external partner relationships, SLAs, and security protocols
• Field testing and pilot operations
• Scoring operations for constructed-response and performance-based assessments
• Project timelines, risk identification, and stakeholder communication
• Core assessment documentation — specs, blueprints, training materials, technical reports

What Helps You Succeed

• You thrive in ambiguity and are energized by building from scratch
• You balance psychometric rigor with startup pragmatism — you know when to ship and iterate
• You communicate with unusual clarity across technical and non-technical audiences
• You take ownership without being asked and surface risks early
• You're excited about AI's role in assessment and know where validation is non-negotiable
• You care deeply about fairness and the learners behind every score

Salary Range: $120K-$180K

Perks

• Competitive salary and equity
• Healthcare for you and your dependents (medical, dental, and vision)
• Parental leave for any full-time employee who has been with us for at least 6 months (birthing, non-birthing, and adoptive parents)
• 15 PTO days per year, a winter break, and 12+ paid company holidays per year
• Remote work days
• Lunch provided in-office daily
• Working with super talented people who want to make a difference in the world

Outsmart is committed to building an equitable and inclusive workplace and is proud to be an equal opportunity employer. We consider all qualified applicants for employment regardless of race, color, religion, gender, gender identity or expression, sexual orientation, national origin, disability, age, veteran status, or any other characteristic protected by applicable law.