Natural Language Measurement Specialist

College Board; New York (NY); Hybrid; USD 88,000 - 145,000; Full time

Benefits offered by this job

Annual bonuses and merit-based raises

Mission-driven workplace

Development and success support

Job summary

College Board’s Learning & Assessment team seeks a Natural Language Specialist to advance how language-based performance is measured in high-stakes assessments. The role sits at the intersection of NLP, large language models, and educational measurement, translating measurement constructs into feedback systems and guiding model development.

You will design studies to establish validity and fairness, collaborate with psychometricians, data scientists, and engineers, and ensure rigorous

Qualifications

• Master’s or PhD in a quantitative field or near completion.
• Strong background in psychometrics, reliability, validity, and measurement theory.
• Experience with NLP, LLMs, and measuring language-based constructs.
• Proficiency in Python and data science libraries.

Responsibilities

• Define and operationalize language-based constructs for automated annotation and feedback generation.
• Apply psychometric principles—reliability, validity, dimensionality, and measurement invariance to LLM-based feedback.
• Design and lead validity studies, including human–machine agreement and fairness analyses.
• Develop methods for detecting and mitigating bias in language-based scores.
• Translate measurement requirements into specifications guiding model development and evaluation.
• Contribute to prompt design, fine-tuning, and evaluation of LLM-based systems.
• Build evaluation frameworks connecting model behavior to measurement outcomes.
• Collaborate with senior team members to translate findings into production systems.
• Implement high-quality, maintainable code for model development and evaluation.
• Prepare datasets and support reproducible data processing workflows.
• Partner with engineers to integrate feedback models into scalable systems.
• Collaborate with psychometricians, data scientists, and engineers.

Skills

Measurement theory

NLP / LLM expertise

Python programming

Data analysis

Education

Master’s or PhD in Psychometrics / Educational Measurement / Quantitative Psychology / Statistics / Data Science

Tools

Python

NumPy

pandas

PyTorch

Job description

Natural Language Specialist, Educational Measurement & AI College Board – Learning & Assessment

Location: This is a remote role. Candidates who live near CB offices have the option of being fully remote or hybrid (Tuesday and Wednesday in office). All CB employees are required to occasionally travel to meet in person for business purposes.

Role Type: This is a full-time position

About the Team

The Automated Scoring team provides critical insights and tools to support the design, delivery, and continuous improvement of digital assessments. We operate at the intersection of educational measurement, data science, and emerging AI technologies. Our work spans the measurement of language-based constructs, the development of large language model (LLM) systems for feedback generation and annotation, and research to ensure the validity, fairness, and reliability of our systems. We are a collaborative, mission-driven team that values both psychometric rigor and technical expertise. We combine modern machine learning approaches, including large language models, with strong measurement principles to create scalable, trustworthy solutions that expand opportunity for students.

About the Opportunity

As a Natural Language Specialist, you will help define and advance how language-based performance is measured in high-stakes educational settings. This role sits at the intersection of natural language processing, large language models, and educational measurement, and is ideal for someone who pairs strong measurement training with a working knowledge of modern AI. You will translate measurement constructs into LLM-based feedback and annotation systems, design and conduct the studies that establish their validity, fairness, and reliability, and ensure that what our models produce holds up to rigorous psychometric standards. You will serve as the measurement authority for cross-functional partners—psychometricians, engineers, and data scientists—shaping how language-based constructs are defined, evaluated, and applied across the team's portfolio of systems. Your primary lens will be measurement: defining what feedback and annotations mean, evidencing that they mean it, and improving them over time.

Natural Language Measurement & Psychometrics (40%)

• Define and operationalize language-based constructs for automated annotation and feedback generation


• Apply psychometric principles—reliability, validity, dimensionality, and measurement invariance to LLM-based feedback and annotation systems


• Design and lead validity studies, including human–machine agreement, rater comparison, and fairness analyses across subgroups


• Develop and apply methods for detecting and mitigating bias in language-based scores


• Establish/Recommend annotation guidelines, feedback quality criteria, and standards for acceptable model performance


• Translate measurement requirements into specifications that guide model development and evaluation

LLM & AI Development (30%)

• Contribute to prompt design, fine-tuning, and evaluation of LLM-based feedback and annotation systems


• Develop and refine machine learning models for measuring language-based constructs


• Build evaluation frameworks that connect model behavior to measurement outcomes


• Collaborate with senior team members to translate measurement findings into production systems


• Implement high-quality, maintainable code for model development and evaluation

Research & Validation (10%)

• Lead and contribute to research studies that evaluate model performance and support assessment validity


• Apply statistical and psychometric methods to analyze results and inform model improvements


• Document methodologies and findings in a clear and rigorous manner


• Stay current with advances in educational measurement, NLP, and learning science

Data Engineering & Pipelines (10%)

• Prepare and curate datasets for measurement studies and model evaluation


• Support reproducible data processing workflows for training, evaluation, and monitoring


• Partner with engineers to integrate feedback and annotation models into scalable systems

Team Operations & Collaboration (10%)

• Collaborate closely with psychometricians, data scientists, and engineers


• Contribute to documentation, methodological standards, and team best practices


• Participate in peer reviews and knowledge sharing


• Actively raise the measurement literacy of the broader team—mentoring junior ICs, providing technical feedback on colleagues' work, and building shared standards

About You

• You bring strong measurement training and a genuine interest in how modern AI can be used to measure language-based performance. You are excited about applying psychometric rigor to large language models in high-stakes educational settings.

You have:

• AMaster’sor PhD (or near completion) in a quantitative field such as Psychometrics, Educational Measurement, Quantitative Psychology, Statistics, Data Science, or a related discipline (measurement-focused training strongly valued)


• A solid foundation in measurement theory, including reliability, validity, and fairness; familiarity with IRT, generalizability theory, or related frameworks is highly desirable


• Experience analyzing language or text data, with exposure to NLP or large language models


• Programming skills in Python and familiarity with data science libraries (e.g., pandas, NumPy,PyTorch)


• Demonstrated ability to conduct rigorous research (e.g., thesis, publications, or applied research projects)


• Strong skills in statistical analysis and experimental design


• Familiarity with working with structured and unstructured data


• Strong attention to detail and a commitment to producing high-quality, reproducible work


• The ability to travel 5-10 times a year to College Board offices or on behalf of College Board business.

You are:

• Curious and eager to learn new tools, methods, and domains


• Thoughtful about the implications of AI systems, including fairness and validity


• Able to communicate technical and measurement concepts clearly to diverse audiences


• Comfortable working in a collaborative, cross-functional environment


• Motivated by mission-driven work in education

All roles at College Board require:

• Authorization to work in the United States


• A passion for expanding educational and career opportunities and mission-driven work


• Curiosity and enthusiasm for emerging technologies, with a willingness to experiment with and adopt new AI‑driven solutions and comfort with learning and applying new digital tools independently and proactively.


• Clear and concise communication skills, written and verbal


• A learner's mindset and a commitment to growth: welcoming diverse perspectives, giving and receiving timely, respectful feedback, and continuously improving through iterative learning and user input.


• A drive for impact and excellence: solving complex problems, making data-informed decisions, prioritizing what matters most, and continuously improving through learning, user input, and external benchmarking.


• A collaborative and empathetic approach: working across differences, fostering trust, and contributing to a culture of shared success.

What We Offer

At College Board, we offer more than a paycheck- we provide a meaningful career, a supportive team, and a comprehensive package designed to help you thrive. We’re a self-sustaining nonprofit that believes in fair and competitive compensation grounded in your qualifications, experience, impact, and the market. A Thoughtful Approach to Compensation The hiring range for this role is $88,000–$145,000. Your exact salary will depend on your location, experience, and how your background compares to others in similar roles at the College Board. We aim to make our best offer upfront, rooted in fairness, transparency, and market data. We adjust salaries by location to ensure fairness, no matter where you live. You’ll have open, transparent conversations about compensation, benefits, and what it’s like to work at College Board throughout your hiring process. Check out our careers page for more. College Board reaches more than 7 million students a year, helping them navigate the path from high school to college and career. We’re a mission-driven, not-for-profit membership organization dedicated to excellence in education. Founded 125 years ago, we are committed to clearing a path for all students to own their future. We pioneered programs like the SAT® and AP® to expand opportunities for students and help them develop the skills they need. Our BigFuture® program helps students plan for college, pay for college, and explore careers. Learn more at cb.org. Clearing a path for all students to own their future.

At College Board, our work is guided by four Operating Principles, and we seek team members who not only align with these principles but actively live them out in their day-to-day work. Prioritize and Simplify–We focus on what matters most, reduce complexity, and move quickly when needed. Say and Receive–We give and receive feedback candidly and kindly, welcoming growth and healthy debate. Go for Greatness–We pursue excellence using data, iteration, and bold thinking to raise the bar. Lead as One College Board–We build a culture of trust, inclusion, and shared responsibility for long-term impact.

In addition to a competitive salary and benefits, we offer:

• Annual bonuses and opportunities for merit-based raises and promotions


• A mission-driven workplace where your impact matters


• A team that invests in your development and success

College Board is proud to be an equal opportunity employer. We’re committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, or Veteran status.