Remote Cybersecurity AI Evaluator & Rubric Reviewer

AuraOne; United States; On-site; USD 55,000 - 96,000; Full time

Benefits offered by this job

Remote work

Contractor role

Job summary

AuraOne is seeking a Cybersecurity AI Evaluator for remote evaluation. You will review cybersecurity AI prompts and responses, label edge cases, and provide structured feedback to retrain the modeling team.

As a contractor, you will compare paired outputs, assign severity tags, and calibrate against gold-standard examples weekly, helping AuraOne build auditable labels, rationales, and regression cases for the human data program.

Qualifications

• Experience with cybersecurity AI evaluation or adjacent content for Cybersecurity AI Evaluator work.
• Clear written reasoning that names the issue and the rubric clause being applied.
• Reliable async availability for at least 10 hours per week.
• Attention to detail and ability to flag prompt problems.

Responsibilities

• Evaluate cybersecurity AI evaluation model outputs against a versioned rubric and assign severity tags.
• Compare paired responses and select the stronger answer with written rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them back to the program team for rubric updates.
• Maintain reviewer-quality scores by calibrating against gold-standard examples weekly.
• Document recurring failure modes for the next training run.

Skills

Model output evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

Cybersecurity AI evaluation

Cybersecurity

AI evaluation

Rubric writing

Expert review

Job description

AuraOne is seeking a Cybersecurity AI Evaluator for remote evaluation. You will review cybersecurity AI prompts and responses, label edge cases, and provide structured feedback to retrain the modeling team.

As a contractor, you will compare paired outputs, assign severity tags, and calibrate against gold-standard examples weekly, helping AuraOne build auditable labels, rationales, and regression cases for the human data program.