Proctor Quality & Learning Lead (Global Calibration)

Duolingo

Pittsburgh (Allegheny County)

On-site

USD 140,000 - 195,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Duolingo is seeking a Proctor Quality and Learning Manager to own the end-to-end quality standard for the DET proctoring network. You will define correct decision criteria, run calibration cycles, and build a live training program with monthly interventions and updated guidance.

You will partner with Psychometrics, Security, Operations, Product, and Engineering to ensure accuracy, auditable outcomes, and robust training as technologies and threats evolve.

Qualifications

  • 5+ years in assessment development, scoring/rater quality, certification, or a field involving complex, consequential judgments; experience aligning reviewers to a shared standard through calibration programs.
  • Adult-learning program experience spanning onboarding, certification, ongoing training, remediation, and effectiveness measurement (beyond one-off materials).
  • Exceptional attention to detail and strong judgment in ambiguous cases; ability to produce clear, defensible decisions and guidance.
  • Analytical fluency to interpret quality data, disagreement patterns, and performance trends and translate them into practical interventions.
  • Clear written and verbal communication; ability to turn complex standards into precise, usable guidance and explain difficult calls.
  • Cross-functional leadership without formal authority; experience directing contractors or partners toward clear accountability and outcomes.
  • Demonstrated skill in quality standards and rubric design, building living systems with decision logs, change histories, and controls for auditability.
  • Experience defining quality benchmarks and validation methods for human–technology workflows, including AI-assisted processes.
  • Adaptability — comfort revising standards and programs as risks, products, and technology change.

Responsibilities

  • Own the DET proctoring quality standard — maintain rubrics, decision rules, examples, and guidance that define accurate decisions across routine and difficult edge cases.
  • Lead calibration and adjudication — operate recurring Proctor Certification Test cycles, resolve disputed decisions, identify patterns of disagreement, and ensure outcomes are consistent and auditable.
  • Build and run an ongoing learning program — publish a forward-looking training calendar, deliver monthly interventions, and launch targeted training in response to quality signals, threats, or workflow changes.
  • Measure effectiveness — use calibration results, audits, and performance trends to identify learning needs, evaluate training impact, and decide when to clarify standards, retrain, or update guidance.
  • Guide specialist execution — set expectations for Training Specialists and adjudicators, assign deliverables, review outputs, and operate onboarding and certification that ramp new specialists quickly.
  • Maintain durable sources of truth — keep calibration, adjudication, and training runbooks current so critical processes are documented and auditable.
  • Evolve quality alongside proctoring — translate new threats and workflow changes into clear standards, training, and quality controls in partnership with Psychometrics, Security, Operations, Product, and Engineering.
  • Represent the proctoring craft — bring frontline proctor evidence into product experiments and workflow decisions; assess whether launched changes preserve decision quality.
  • Connect human and technology-assisted quality — establish benchmarks for AI-assisted workflows and validate automated capabilities against real sessions before deployment.

Skills

Attention to detail
Analytical thinking
Clear communication
Cross-functional collaboration
Decision making under ambiguity
Quality standards development

Tools

Learning Management Systems
Assessment platforms
Quality-audit tools
Operational analytics

Job description

Duolingo is seeking a Proctor Quality and Learning Manager to own the end-to-end quality standard for the DET proctoring network. You will define correct decision criteria, run calibration cycles, and build a live training program with monthly interventions and updated guidance.

You will partner with Psychometrics, Security, Operations, Product, and Engineering to ensure accuracy, auditable outcomes, and robust training as technologies and threats evolve.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Proctor Quality & Learning Leader
Proctor Quality & Learning Leader

Duolingo • Pennsylvania

On-site
USD 143,000 - 193,000
Proctor Quality and Learning Manager, Scaling Operations
Proctor Quality and Learning Manager, Scaling Operations

Duolingo • Pennsylvania

On-site
USD 143,000 - 193,000
Proctor Quality and Learning Manager, Scaling Operations
Proctor Quality and Learning Manager, Scaling Operations

Duolingo • Pittsburgh

On-site
USD 140,000 - 195,000
Senior PM: DET UX & AI Proctoring
Senior PM: DET UX & AI Proctoring

The Briefed • New York (NY)

On-site
USD 183,000 - 247,000
Competitive base salary and equity
Medical, dental, and vision insurance
Generous time-off & development budget
Senior Product Manager — Scale a Global English Test
Senior Product Manager — Scale a Global English Test

Duolingo • Pennsylvania

On-site
USD 183,000 - 247,000
Senior Product Manager: English Test & Global Growth
Senior Product Manager: English Test & Global Growth

EngineersOfAI • New York (NY)

On-site
USD 150,000 - 190,000
Senior Product Manager, Global English Test
Senior Product Manager, Global English Test

EngineersOfAI • Pittsburgh

On-site
USD 120,000 - 150,000
Senior Product Manager - DET User Experience & AI
Senior Product Manager - DET User Experience & AI

The Briefed • New York (NY)

On-site
USD 183,000 - 247,000
Competitive base salary and equity
Medical, dental, and vision insurance
Generous time-off & development budget
Head of Data Quality & Standards
Head of Data Quality & Standards

Reinforced Learning • Town of Tonawanda (NY)

On-site
USD 120,000 - 180,000
Staff TPM — Lead High-Impact Cross-Functional Programs
Staff TPM — Lead High-Impact Cross-Functional Programs

EngineersOfAI • Pittsburgh

On-site
USD 140,000 - 210,000