Remote AI Dialogue & Evaluation Specialist

AI Trainer Jobs; United States; Remote; USD 28,000 - 50,000; Full time

Job summary

AI Trainer Jobs is seeking an AI Agent Interaction Specialist - Dialogue & Conversation Evaluation for a remote contractor role. You will review ai agent interaction prompts and responses against AuraOne's quality rubric, label edge cases, and provide structured feedback to retrain the model.

You will judge model behavior on text, images, and documents, including mixed-input tasks. Responsibilities include evaluating model outputs against a versioned rubric, selecting the stronger response with

Qualifications

• Prior evaluation, annotation, or human-rater experience on ai agent interaction evaluation or adjacent content for AI Agent Interaction Specialist - Dialogue & Conversation Evaluation work.
• Clear written reasoning that names the issue and the rubric clause being applied.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Evaluate ai agent interaction evaluation model outputs against a versioned rubric and assign severity tags for AI Agent Interaction Specialist - Dialogue & Conversation Evaluation assignments.
• Compare paired responses and pick the stronger answer with a written rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them back to the program team for rubric updates.
• Audit a 50-row batch for rubric consistency and report drift to the program lead.
• Propose a rubric clarification after spotting a recurring failure mode.

Skills

Annotation experience

Inter-rater agreement

Attention to detail

Structured reasoning

Rubric-based evaluation

Job description

AI Trainer Jobs is seeking an AI Agent Interaction Specialist - Dialogue & Conversation Evaluation for a remote contractor role. You will review ai agent interaction prompts and responses against AuraOne's quality rubric, label edge cases, and provide structured feedback to retrain the model.

You will judge model behavior on text, images, and documents, including mixed-input tasks. Responsibilities include evaluating model outputs against a versioned rubric, selecting the stronger response with