Remote CTF Challenge Security Evaluator (Contractor)

AuraOne; United States; On-site; USD 34,440 - 61,992; Full time

Job summary

AuraOne is seeking a remote CTF Challenge Security Evaluator to help assess security evaluation prompts and responses against a quality rubric. Role involves reviewing paired outputs, labeling edge cases, and providing structured feedback for retraining the model.

Responsibilities include evaluating outputs, selecting stronger answers with rationale, tagging policy violations, and documenting failure modes for ongoing rubric improvements. Independent contractor with US eligibility is required.

Qualifications

• Prior evaluation, annotation, or human-rater experience on ctf challenge security evaluation or adjacent content for CTF Challenge Security Evaluator work.
• Clear written reasoning that names the issue and the rubric clause being applied.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Evaluate ctf challenge security evaluation model outputs against a versioned rubric and assign severity tags for CTF Challenge Security Evaluator assignments.
• Compare paired responses and pick the stronger answer with a written rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them back to the program team for rubric updates.
• Maintain reviewer-quality scores by calibrating against gold-standard examples each week.
• Document recurring failure modes so the modeling team can target them in the next training run.

Skills

Model output evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

CTF Challenge Security evaluation

Security review

Adversarial testing

Threat modeling

Challenge

Job description

AuraOne is seeking a remote CTF Challenge Security Evaluator to help assess security evaluation prompts and responses against a quality rubric. Role involves reviewing paired outputs, labeling edge cases, and providing structured feedback for retraining the model.

Responsibilities include evaluating outputs, selecting stronger answers with rationale, tagging policy violations, and documenting failure modes for ongoing rubric improvements. Independent contractor with US eligibility is required.