Remote Cloud Security Evaluator: Model Feedback & Review

AuraOne; United States; On-site; USD 55,104 - 96,432; Full time

Job summary

AuraOne is seeking a Cloud Security Evaluator for a remote contractor role to review evaluation prompts and responses against our quality rubric. You will compare paired outputs, assign severity labels, and write structured feedback the modeling team can use to retrain.

Responsibilities include evaluating model outputs, selecting the stronger answer with a written rationale, tagging hallucinations and unsafe content, and documenting recurring failure modes for rubric updates.

Qualifications

• Prior evaluation, annotation, or human-rater experience on cloud security evaluation or adjacent content.
• Clear written reasoning that names the issue and the rubric clause being applied.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Evaluate model outputs against a versioned rubric and assign severity tags.
• Compare paired responses and provide rationale for the stronger answer.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them for rubric updates.
• Maintain reviewer-quality scores by calibrating against gold-standard examples.
• Document recurring failure modes for downstream training.

Skills

Model output evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

Cloud Security evaluation

Security review

Adversarial testing

Threat modeling

Cloud

Security

Job description

AuraOne is seeking a Cloud Security Evaluator for a remote contractor role to review evaluation prompts and responses against our quality rubric. You will compare paired outputs, assign severity labels, and write structured feedback the modeling team can use to retrain.

Responsibilities include evaluating model outputs, selecting the stronger answer with a written rationale, tagging hallucinations and unsafe content, and documenting recurring failure modes for rubric updates.