Annotation Data Engineer

Envision Technology Solutions; United States; On-site; USD 90,000 - 150,000; Full time

Job summary

Envision Technology Solutions is seeking a detail-oriented Quality & Data Operations Specialist to support AI data quality programs in the United States. You will contribute to the development of metrics, guidelines, defect taxonomies, and audit processes for large-scale annotation projects to ensure data reliability and model performance.

You will collaborate across annotation operations, engineering, and AI teams to monitor quality, drive corrective actions, and validate dataset releases while

Qualifications

• Experience in annotation quality and data operations for AI/ML
• Experience developing quality metrics and measurement systems
• Experience with large-scale annotation programs
• Experience evaluating generated captions and foundation model training datasets
• Ability to define sampling plans, acceptance criteria, defect taxonomies and audit processes
• Ability to analyze quality trends and drive corrective actions

Skills

Annotation quality

Quality metrics

Large-scale programs

Caption evaluation

Sampling plans

Root-cause analysis

Quality standards

Metrics reporting

Audits

Quality trends

Dataset releases

Calibration sessions

Attention to detail

Stakeholder communication

Job description

• Demonstrated experience in annotation quality, machine learning data operations, quality engineering, or data quality supporting physical AI, robotics, computer vision, or machine learning programs.
• Demonstrated experience developing quality metrics and performance measurement systems.
• Demonstrated experience working with large-scale annotation programs.
• Demonstrated experience evaluating generated captions, reasoning outputs, or foundation model training datasets.
• Experience defining annotation instructions, sampling plans, acceptance criteria, defect taxonomies, and audit processes.
• Ability to analyze quality trends, identify root causes, and coordinate corrective actions.
• Working knowledge of SQL, Python, reporting, or data analysis tools used for quality measurement.
• Define and maintain annotation quality standards, guidelines, defect categories, and acceptance criteria.
• Develop metrics and reporting for accuracy, consistency, completeness, rework, throughput, and reviewer agreement.
• Design sampling, audit, escalation, and root-cause analysis processes for large-scale annotation programs.
• Evaluate generated captions, reasoning outputs, and foundation model training datasets against defined quality criteria.
• Analyze quality trends and partner with annotation operations, engineering, and AI teams on corrective actions.
• Validate dataset releases and maintain traceable quality records, issue logs, and improvement plans.
• Support calibration sessions, reviewer alignment, and continuous improvement of annotation workflows.
• Generic Managerial Skills, If Any
• Strong attention to detail, analytical thinking, process discipline, stakeholder communication, and ability to influence quality improvements across teams.