LLM Evaluation Architect | Prompt & Rubric Design

Crossing Hurdles; Philippines; On-site; PHP 669,600 - 1,004,400; Full time

Job summary

Crossing Hurdles is seeking a meticulous evaluator to develop adversarial multi-turn conversations and evaluation rubrics that align with project specifications. You will author clear rubrics and measure model responses against target behaviors, iterating with frontier LLMs to increase difficulty.

The role requires strong written English, experience in AI data environments, and sharp attention to detail. As part of the project team, you will produce comprehensive task packages, including

Qualifications

• Exceptional written English with clarity, precision, and strong structure.
• Experience in AI human data environments (RLHF, SFT, evaluations, or prompt engineering).
• Deep familiarity with large language models and ability to identify common failure patterns.
• Ability to work autonomously, interpreting and executing complex specifications.
• Proven critical thinking and meticulous attention to detail.
• Experience designing evaluation items or rubrics is advantageous.

Responsibilities

• Develop complex, adversarial multi-turn conversations and task-based scenarios aligned with project specifications.
• Author clear, precise evaluation rubrics to assess model responses against defined behavioral targets.
• Iteratively test conversations and tasks against frontier LLMs, escalating difficulty and nuance.
• Deliver comprehensive task packages, including transcripts, target behaviors, and supporting rationale.
• Validate LLM outputs, documenting model strengths and failure modes relative to specifications.
• Maintain calibration with team leads and quality control contacts as project requirements evolve.

Skills

Exceptional English

RLHF / SFT experience

LLM familiarity

Autonomous work

Critical thinking

Rubric design

Job description

Crossing Hurdles is seeking a meticulous evaluator to develop adversarial multi-turn conversations and evaluation rubrics that align with project specifications. You will author clear rubrics and measure model responses against target behaviors, iterating with frontier LLMs to increase difficulty.

The role requires strong written English, experience in AI data environments, and sharp attention to detail. As part of the project team, you will produce comprehensive task packages, including