AI Data Quality Specialist — Large-Scale Annotation

Covetus; Dallas (TX); On-site; USD 70,000 - 110,000; Full time

Job summary

Covetus is seeking a Data Quality and Annotation Specialist to strengthen annotation quality across large‑scale ML datasets. You will define instructions, sampling plans, and audit processes to ensure accuracy and consistency.

You will develop metrics for accuracy, throughput, and reviewer agreement, analyze quality trends, and coordinate corrective actions with engineering and AI teams. Strong attention to detail and communication are essential.

Qualifications

• Experience defining annotation instructions, sampling plans, acceptance criteria, defect taxonomies, and audit processes.
• Ability to analyze quality trends, identify root causes, and coordinate corrective actions.
• Working knowledge of SQL, Python, reporting, or data analysis tools used for quality measurement.

Responsibilities

• Define annotation instructions, sampling plans, acceptance criteria, defect taxonomies and audit processes.
• Develop metrics and reporting for accuracy, consistency, completeness, rework, throughput, and reviewer agreement.
• Design sampling, audit, escalation, and root-cause analysis for large-scale annotation programs.
• Evaluate captions, reasoning outputs, and foundation model training datasets against quality criteria.
• Analyze quality trends and coordinate corrective actions with annotation operations, engineering, and AI teams.
• Validate dataset releases and maintain traceable quality records, issue logs and improvement plans.
• Support calibration sessions, reviewer alignment, and continuous improvement of workflows.
• Strong attention to detail and stakeholder communication to drive improvements across teams.

Skills

Annotation quality

ML data operations

Quality engineering

Data quality

Quality metrics

Large-scale annotation

Caption evaluation

Defect taxonomy

Audit processes

Root cause analysis

Acceptance criteria

Reviewer agreement

Sampling plans

Traceability

Stakeholder communication

Process discipline

Tools

SQL

Python

Reporting tools

Job description

Covetus is seeking a Data Quality and Annotation Specialist to strengthen annotation quality across large‑scale ML datasets. You will define instructions, sampling plans, and audit processes to ensure accuracy and consistency.

You will develop metrics for accuracy, throughput, and reviewer agreement, analyze quality trends, and coordinate corrective actions with engineering and AI teams. Strong attention to detail and communication are essential.