Annotation Data Engineers (1157993)

The Judge Group; Dallas (TX); Remote; USD 90,000 - 100,000; Full time

Benefits offered by this job

Medical, dental, and vision

Retirement plan with company contrib.

Paid time off and holidays

Career development

Job summary

The Judge Group is seeking an Annotation Data Engineer to support large-scale annotation programs. You will define quality standards, develop metrics, and oversee audits of captions and foundation model data across remote teams.

This full-time role allows CST or EST-based coordination with cross-functional AI and engineering teams. You will evaluate generated captions and outputs, define sampling plans and defect taxonomies, and ensure traceable quality records.

Qualifications

• Demonstrated experience in annotation quality, ML data operations, quality engineering, or data quality.
• Experience developing quality metrics and performance measurement systems.
• Experience working with large-scale annotation programs.

Responsibilities

• Define and maintain annotation quality standards, guidelines, defect categories, and acceptance criteria.
• Develop metrics and reporting for accuracy, consistency, completeness, rework, throughput, and reviewer agreement.
• Design sampling, audit, escalation, and root-cause analysis processes for large-scale annotation programs.

Skills

Annotation quality

Quality metrics

Large-scale annotation

Caption evaluation

Annotation instructions

Quality trend analysis

SQL

Python

Tools

SQL

Python

Reporting tools

Job description

Location

Dallas, TX, USA

Salary

$90,000.00 USD Annually - $100,000.00 USD Annually

Description

Annotation Data Engineers

Remote (CST or EST)

Fulltime

Must Have Technical/Functional Skills

• Demonstrated experience in annotation quality, machine learning data operations, quality engineering, or data quality supporting physical AI, robotics, computer vision, or machine learning programs.


• Demonstrated experience developing quality metrics and performance measurement systems.


• Demonstrated experience working with large-scale annotation programs.


• Demonstrated experience evaluating generated captions, reasoning outputs, or foundation model training datasets.


• Experience defining annotation instructions, sampling plans, acceptance criteria, defect taxonomies, and audit processes.


• Ability to analyze quality trends, identify root causes, and coordinate corrective actions.


• Working knowledge of SQL, Python, reporting, or data analysis tools used for quality measurement.

Roles & Responsibilities

• Define and maintain annotation quality standards, guidelines, defect categories, and acceptance criteria.


• Develop metrics and reporting for accuracy, consistency, completeness, rework, throughput, and reviewer agreement.


• Design sampling, audit, escalation, and root-cause analysis processes for large-scale annotation programs.


• Evaluate generated captions, reasoning outputs, and foundation model training datasets against defined quality criteria.


• Analyze quality trends and partner with annotation operations, engineering, and AI teams on corrective actions.


• Validate dataset releases and maintain traceable quality records, issue logs, and improvement plans.


• Support calibration sessions, reviewer alignment, and continuous improvement of annotation workflows.

Comprehensive benefits package including medical, dental, and vision coverage, retirement savings plan with company contribution, paid time off, holidays, and opportunities for professional growth and career development. Additional benefits and perks will be discussed during the interview process.

Contact: [e-mail]