Remote AI Evaluation & Calibration Specialist

Crossing Hurdles; Philippines; On-site; PHP 279,000 - 502,200; Full time

Job summary

Crossing Hurdles seeks meticulous contributors to craft and evaluate AI conversation drafts. You will write detailed, task-based conversations, align rubrics with project specs, and iteratively refine outputs against frontier language models.

Applicants should have native-level English, experience in data annotation or AI evaluation, and strong analytical skills. You will work independently, delivering quality transcripts and evidence while adhering to evolving guidelines.

Qualifications

• Native-level written English with exceptional clarity, structure and attention to detail.
• Experience in data annotation, RLHF, SFT, evaluation, or prompt engineering for AI systems.
• Knowledge of frontier LLM behaviors and common model failure patterns.
• Ability to interpret and apply highly detailed specifications without supervision.
• Strong critical thinking and analytical skills in writing-heavy or analysis-heavy domains.

Responsibilities

• Author detailed, task-based multi-turn conversations and rubrics aligned with project specifications.
• Test and refine conversation drafts against frontier large language models, iterating to meet quality and difficulty requirements.
• Deliver comprehensive evaluation assets including transcripts, target behaviors, binary rubrics, and supporting evidence.
• Ensure strict fidelity to evolving project specs while maintaining high throughput and attention to detail.
• Validate and calibrate outputs with team leads and quality control as guidelines change.
• Work independently and consistently, meeting expected output rates for deliverable completion.

Skills

Native English writing

Data annotation

RLHF evaluation

Analytical thinking

Job description

Crossing Hurdles seeks meticulous contributors to craft and evaluate AI conversation drafts. You will write detailed, task-based conversations, align rubrics with project specs, and iteratively refine outputs against frontier language models.

Applicants should have native-level English, experience in data annotation or AI evaluation, and strong analytical skills. You will work independently, delivering quality transcripts and evidence while adhering to evolving guidelines.