Senior Data Annotator - Remote AI Training & Evaluation

Rex.zone; Warszawa; On-site; PLN 60,000 - 90,000; Full time

Job summary

Rex.zone in Warsaw (Remote) is seeking senior data annotators to create and validate high-quality training data for AI/ML systems. You will support LLM training pipelines through data labeling, QA evaluation, prompt evaluation, and RLHF-style ranking to improve model behavior and evaluation quality.

You will annotate NLP, vision, and safety datasets, perform LLM evaluation tasks, conduct QA audits, and help refine guidelines with edge-case documentation while thriving in a remote,

Qualifications

• Experience in data annotation/data labeling with strong quality outcomes.
• Excellent annotation guidelines compliance and consistency on edge cases.
• Experience with LLM evaluation, prompt evaluation, or RLHF-style ranking.
• Strong written documentation and feedback skills.
• Comfort working remotely with throughput, accuracy, and audit expectations.

Responsibilities

• Annotate and review datasets across NLP, computer vision, and content safety domains.
• Perform LLM evaluation tasks such as pairwise ranking, multi-turn conversation grading, and RLHF-style preference data creation.
• Execute QA evaluation via audits, defect pattern reporting, and inter-annotator agreement checks.
• Conduct named entity recognition (NER), taxonomy tagging, and consistent label application.
• Evaluate prompts and model outputs for helpfulness, harmlessness, factuality, and policy compliance.
• Collaborate with leads to refine guidelines, document edge cases, and improve calibration.

Skills

Data annotation

Guidelines compliance

LLM evaluation

Documentation

Remote work

Job description

Rex.zone in Warsaw (Remote) is seeking senior data annotators to create and validate high-quality training data for AI/ML systems. You will support LLM training pipelines through data labeling, QA evaluation, prompt evaluation, and RLHF-style ranking to improve model behavior and evaluation quality.

You will annotate NLP, vision, and safety datasets, perform LLM evaluation tasks, conduct QA audits, and help refine guidelines with edge-case documentation while thriving in a remote,