Validation Engineer - Data Annotator

Zobility; Dearborn (MI); On-site; USD 45,000 - 75,000; Full time

Job summary

Zobility is seeking a detail-oriented Data Annotation professional to enhance chatbot quality through a structured human-in-the-loop feedback system in Dearborn, MI. You will review conversations, apply labeling guidelines, and document decisions with clear rationales to ensure accuracy and reliability of AI outputs.

Ideal candidates will have experience labeling conversational AI data, strong analytical skills, and fluency in English. Automotive industry familiarity is a plus.

Qualifications

• Analytical, communications, and data annotation for voice and text.
• Experience labeling conversational AI data (text and audio).
• Automotive industry experience preferred.
• English fluency with strong reading and writing skills.

Responsibilities

• Improve chatbot response quality by human-evaluated feedback loop.
• Review conversations including turns, model responses, citations, and metadata in client tools.
• Apply labeling guidelines, taxonomies, decision rules, and complete labels with rationales.
• Evaluate chatbot performance across defined dimensions.
• Identify content needing expert review or is ambiguous or safety-relevant.
• Maintain records in client systems and provide weekly reports.
• Suggest improvements to guidelines and taxonomy with client approval.

Skills

Analytical

Communication

Data Annotation

English Fluency

Attention to Detail

Job description

Responsibilities

• Improve the quality of the chatbot responses by creating a feedback loop where AI responses are judged by humans.
• Review chatbot conversations and related artifacts including individual turns, model responses, retrieved content, citations, and metadata within Client-approved tools.
• Apply Client-approved labeling guidelines, taxonomies, and decision rules, completing all required labels, rationales, confidence assessments, and ambiguity indicators.
• Evaluate chatbot performance across dimensions defined by Client, including intent, task completion, correctness, completeness, relevance, groundedness, source attribution, response quality, safety, policy compliance, escalation need, and failure category.
• Identify and elevate content that is ambiguous, sensitive, safety-relevant, outside the current guidelines, or requires subject-matter-expert review.
• Labelers must not infer or create standards where approved guidance does not exist.
• Participate in onboarding, calibration, refresher training, quality reviews, and adjudication sessions to maintain consistent labeling decisions.
• Conduct first-level quality checks on assigned work and correct errors identified through Client or supplier quality reviews.
• Maintain complete and accurate records of completed annotations, quality results, exceptions, escalations, and open issues in Client-designated systems.
• Provide weekly reports summarizing completed volume, productivity, quality results, backlog, open questions, risks, guideline gaps, and required actions.
• Recommend and document improvements to the annotation guidelines and taxonomy based on recurring ambiguities, labeling patterns, and adjudication outcomes. Changes may be implemented only after Client approval.

Requirements

• Analytical, Communications, Data Annotation - Voice and Text
• Experience labeling conversational AI data, including text and audio.
• Prior experience in the automotive industry is preferred.
• Professional fluency in English, with strong reading comprehension and written communication skills. Demonstrated ability to interpret and consistently apply detailed guidelines, taxonomies, and decision rules.
• Strong analytical judgment and attention to detail, including the ability to identify factual errors, incomplete responses, unsupported claims, policy concerns, and ambiguous cases.
• Ability to document labeling decisions and provide clear, concise rationales.
• Ability to work accurately on repetitive tasks while meeting quality, productivity, and turnaround expectations.
• Ability to participate in calibration sessions, incorporate feedback, and adjust decisions as guidelines evolve.
• Ability to prepare clear weekly reports summarizing labeling volume, productivity, quality results, progress, and open issues.