Data Quality Lead Data Operations Remote · United States · Full-time

Humxn; United States; Remote; USD 90,000 - 130,000; Full time

Job summary

Humxn is seeking a role focused on defining and enforcing quality for ML datasets. You will design rubrics, guidelines, and acceptance criteria so experts apply them consistently, run qualification and review processes, and measure agreement and error rates per batch.

You will investigate quality issues and drive root-cause fixes, collaborating with buyers' research teams to align on what “good” means for their models.

Qualifications

• Experience running annotation, evaluation or data-quality programs for ML.
• Comfort with basic statistics for agreement and sampling.
• Clear, precise writing; guidelines are a product.

Responsibilities

• Turn dataset specs into rubrics, guidelines and acceptance criteria experts can apply consistently.
• Run qualification and review processes, and measure agreement and error rates per batch.
• Investigate quality issues and fix root causes from unclear guidelines to wrong expert match.
• Partner with buyers' research teams on what “good” means for their models.

Skills

Annotation programs

Statistics basics

Technical writing

Job description

About the role

"Verified" has to mean something measurable. You'll design how we define and check quality for each dataset: rubrics, gold items, second-expert review and agreement metrics, and you'll make sure buyers can see the evidence.

What you'll do

• Turn dataset specs into rubrics, guidelines and acceptance criteria experts can apply consistently.
• Run qualification and review processes, and measure agreement and error rates per batch.
• Investigate quality issues and fix their root causes, from unclear guidelines to the wrong expert match.
• Partner with buyers' research teams on what "good" means for their models.

What we're looking for

• Experience running annotation, evaluation or data-quality programs for ML.
• Comfort with basic statistics for agreement and sampling.
• Clear, precise writing; guidelines are a product.

Nice to have

• Experience with RLHF, preference or evaluation data.