Software Engineer — Agentic Search Systems

AI Trainer Jobs; United States; Remote; USD 110,000 - 207,000; Full time

Benefits offered by this job

Remote

Job summary

Agentic Search Systems is seeking a Software Engineer for a remote evaluation track. Review prompts and responses against AuraOne's rubric, label edge cases, and draft structured feedback for retraining the model.

Responsibilities include comparing outputs, selecting stronger answers with rationale, tagging issues, and routing ambiguous prompts back to the team for rubric updates. This is a contractor, remote role with US eligibility.

Qualifications

• Experience evaluating model outputs against rubrics.
• Ability to identify edge cases and provide structured feedback.
• Comfort labeling unsafe content and policy adherence.

Responsibilities

• Evaluate model outputs against rubric and assign severity tags.
• Compare paired responses and write rationale.
• Label hallucinations and policy violations with tags.
• Capture ambiguous prompts and suggest rubric updates.

Skills

Evaluation experience

Annotation

Clear reasoning

Attention to detail

Async availability

Job description

Software Engineer — Agentic Search Systems is a remote evaluation track for reviewing software engineer agentic search systems evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and write the kind of structured feedback the modeling team can use to retrain.

Category: Search, Web & Browser Agents · Pay: $80–$150 / hr · Location: Remote — US-eligible · Contractor

Software Engineer — Agentic Search Systems is a remote evaluation track for reviewing software engineer agentic search systems evaluation prompts and responses against AuraOne's quality rubric.

About the role

Software Engineer — Agentic Search Systems is a remote evaluation track for reviewing software engineer agentic search systems evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and write the kind of structured feedback the modeling team can use to retrain.
AI data reviewers help turn software engineer agentic search systems evaluation outputs into auditable labels, rationales, and regression cases for AuraOne Human Data.
Review browser automation and multi-step search agents. Check whether sources hold up.

Responsibilities

• Evaluate software engineer agentic search systems evaluation model outputs against a versioned rubric and assign severity tags for Software Engineer — Agentic Search Systems assignments.
• Compare paired responses and pick the stronger answer with a written rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them back to the program team for rubric updates.

Role details

Track Evaluation & annotation Work model Remote · Independent specialist contractor Compensation $80–$150 / hr Eligible from US

What you should bring

• Prior evaluation, annotation, or human-rater experience on software engineer agentic search systems evaluation or adjacent content for Software Engineer — Agentic Search Systems work.
• Comfort applying multi-page rubrics consistently across long batches.
• Clear written reasoning that names the issue and the rubric clause being applied.
• Strong attention to detail and the ability to flag when a prompt itself is the problem.
• Reliable async availability for at least 10 hours per week.

Role signals

Example tasks

• Compare two software engineer agentic search systems evaluation model responses to the same prompt and pick the stronger one with rationale.
• Tag an unsafe response with the correct policy category and severity.
• Audit a 50-row batch for rubric consistency and report drift to the program lead.
• Propose a rubric clarification after spotting a recurring failure mode.

Useful experience

• Background in linguistics, content moderation, or trust & safety review.
• Experience with inter-rater agreement metrics and calibration cycles.
• Domain expertise that lets you spot subject-matter errors automated checks miss.

Compensation and schedule

$80–$150 / hr
Expected arrangement: contractor , with program-defined task volume and review pacing. Placement depends on current program demand and reviewer confirmation.

Skills used in matching

• Model output evaluation
• Rubric-based annotation
• Severity tagging
• Inter-rater calibration
• Software Engineer Agentic Search Systems evaluation