Remote AI Data Reviewer for Agentic Search Systems

AI Trainer Jobs; United States; Remote; USD 110,000 - 207,000; Full time

Benefits offered by this job

Remote

Job summary

Agentic Search Systems is seeking a Software Engineer for a remote evaluation track. Review prompts and responses against AuraOne's rubric, label edge cases, and draft structured feedback for retraining the model.

Responsibilities include comparing outputs, selecting stronger answers with rationale, tagging issues, and routing ambiguous prompts back to the team for rubric updates. This is a contractor, remote role with US eligibility.

Qualifications

• Experience evaluating model outputs against rubrics.
• Ability to identify edge cases and provide structured feedback.
• Comfort labeling unsafe content and policy adherence.

Responsibilities

• Evaluate model outputs against rubric and assign severity tags.
• Compare paired responses and write rationale.
• Label hallucinations and policy violations with tags.
• Capture ambiguous prompts and suggest rubric updates.

Skills

Evaluation experience

Annotation

Clear reasoning

Attention to detail

Async availability

Job description

Agentic Search Systems is seeking a Software Engineer for a remote evaluation track. Review prompts and responses against AuraOne's rubric, label edge cases, and draft structured feedback for retraining the model.

Responsibilities include comparing outputs, selecting stronger answers with rationale, tagging issues, and routing ambiguous prompts back to the team for rubric updates. This is a contractor, remote role with US eligibility.