LLM Annotator – Execution (Tier 2 & Tier 3)

Crescendo Staffing and Business Consulting Inc.; Quezon City; On-site; PHP 334,800 - 558,000; Full time

Job summary

A staffing and consulting firm is seeking an LLM Annotator – Execution (Tier 2 & Tier 3) to perform advanced annotation and quality assurance tasks for Large Language Models (LLMs). The ideal candidate will have at least one year of relevant experience and strong analytical skills. Key responsibilities include executing complex annotation tasks, evaluating LLM outputs, and collaborating on quality assurance activities. A Bachelor's degree in a related field is preferred, along with strong attention to detail and the ability to work independently.

Qualifications

• At least 1 year of LLM-related work experience, including annotation, evaluation, prompt review, or AI operations.
• Experience working with NLP concepts and AI safety guidelines is a plus.
• Ability to work independently in a structured, metrics-driven environment.

Responsibilities

• Execute complex text annotation, labeling, and classification tasks.
• Perform LLM output evaluation based on relevance and accuracy.
• Handle ambiguous inputs with sound judgment.
• Support quality assurance activities such as peer reviews and error analysis.
• Maintain productivity and quality metrics aligned with project standards.

Skills

Strong English comprehension

Attention to detail

Ability to follow guidelines

Analytical skills

Education

Bachelor’s degree in Linguistics, Communications, Computer Science, Data Science, Psychology, or a related field

Tools

Annotation tools

AI evaluation systems

Job description

Job Title: LLM Annotator – Execution (Tier 2 & Tier 3)

Function: AI Operations / Data Annotation

Location: Philippines

Employment Type: Full-Time

Experience Level: Tier 2 / Tier 3

Experience Requirement: At least 1 year of LLM-related work experience preferred

Job Summary

The LLM Annotator – Execution (Tier 2 & Tier 3) is responsible for performing advanced annotation, evaluation, and quality assurance tasks that support the training, tuning, and validation of Large Language Models (LLMs). This role executes structured annotation workflows, applies detailed labeling guidelines, and ensures high-quality outputs that meet accuracy, consistency, and compliance standards.

Tier 2 and Tier 3 annotators are expected to demonstrate higher judgment, complexity handling, and quality ownership compared to entry-level roles.

Key Responsibilities

• Execute complex text annotation, labeling, and classification tasks following detailed project guidelines.
• Perform LLM output evaluation, including relevance, accuracy, safety, and instruction adherence.
• Handle edge cases, ambiguous inputs, and nuanced language scenarios with sound judgment.
• Apply consistent annotation decisions across large datasets with minimal supervision.
• Identify and flag quality issues, guideline gaps, or unclear instructions for escalation.
• Support quality assurance (QA) activities such as peer reviews, spot checks, and error analysis.
• Maintain productivity, accuracy, and quality metrics aligned with project standards.
• Document annotation decisions and rationales when required.
• Adhere to data privacy, confidentiality, and security requirements.

Tier-Specific Expectations

Tier 2

• Executes moderately complex annotation tasks independently.
• Demonstrates strong guideline comprehension and consistency.
• Meets productivity and quality benchmarks with minimal rework.
• Escalates unclear cases appropriately.

Tier 3

• Handles highly complex or sensitive annotation tasks.
• Applies advanced judgment to nuanced language, intent, and context.
• Provides informal guidance or calibration support to Tier 1 and Tier 2 annotators.
• Contributes to quality improvement by identifying recurring issues and suggesting refinements.

Qualifications

• Bachelor’s degree in Linguistics, Communications, Computer Science, Data Science, Psychology, or a related field (preferred).
• At least 1 year of LLM-related work experience is preferred, including annotation, evaluation, prompt review, or AI operations.
• Strong English comprehension, writing, and analytical skills.
• Ability to follow detailed guidelines and maintain high accuracy.
• Experience working with annotation tools, internal platforms, or AI evaluation systems (preferred).
• High attention to detail and ability to work on repetitive yet complex tasks.

Preferred Skills

• Prior experience with LLM annotation, model evaluation, RLHF, or content moderation.
• Familiarity with NLP concepts, prompt-response evaluation, or AI safety guidelines.
• Ability to work independently in a structured, metrics-driven environment.
• Experience supporting global AI or data operations teams.