Senior Data Annotation Jobs in Rome

RemoExperts; Roma; Remote; EUR 35,000 - 60,000; Full time

Job summary

RemoExperts in Italy is seeking a Senior Data Annotation professional to support AI/ML training pipelines across NLP and computer vision tasks. You will own complex labeling, RLHF workflows, and QA evaluation to improve model performance.

Collaborate with AI labs, startups, and vendors, maintain guidelines, document decisions, and ensure high-quality datasets delivered on schedule.

Qualifications

• Production experience in data annotation/data labeling or QA evaluation for AI/ML systems.
• Strong understanding of training data quality and guideline-driven labeling.
• Familiarity with RLHF and large language model evaluation concepts (prompt evaluation, preference ranking).
• High attention to detail, consistent decision-making, and comfort with ambiguous edge cases.
• Clear written communication and precise documentation habits.

Responsibilities

• Own and execute complex annotation tasks across NLP and computer vision annotation (e.g., named entity recognition, classification, ranking, span labeling).
• Perform RLHF workflows (preference comparisons, rationale checks, policy-aligned scoring) and prompt evaluation for LLM training pipelines
• Run QA evaluation using sampling plans, inter-annotator agreement checks, and targeted error analysis
• Maintain annotation guidelines compliance by documenting decisions, escalating ambiguous cases, and updating rubrics
• Partner with stakeholders across AI labs, tech startups, BPO teams, and annotation vendors to deliver production-ready datasets on schedule

Skills

Data annotation

QA evaluation

Attention to detail

Documentation

Job description

About The Role

Senior data annotation is a Mid-Senior, full-time remote role focused on training data quality for AI/ML systems—labeling, RLHF, prompt evaluation, and QA evaluation that directly improve model performance.

You will help build reliable LLM training pipelines by applying annotation guidelines compliance, auditing datasets, and refining instructions for human feedback.

What You Will Do

• Own and execute complex annotation tasks across NLP and computer vision annotation (e.g., named entity recognition, classification, ranking, span labeling)
• Perform RLHF workflows (preference comparisons, rationale checks, policy-aligned scoring) and prompt evaluation for LLM training pipelines
• Run QA evaluation using sampling plans, inter-annotator agreement checks, and targeted error analysis
• Maintain annotation guidelines compliance by documenting decisions, escalating ambiguous cases, and updating rubrics
• Partner with stakeholders across AI labs, tech startups, BPO teams, and annotation vendors to deliver production-ready datasets on schedule

Required Qualifications

• Production experience in data annotation/data labeling or QA evaluation for AI/ML systems
• Strong understanding of training data quality and guideline-driven labeling
• Familiarity with RLHF and large language model evaluation concepts (prompt evaluation, preference ranking)
• High attention to detail, consistent decision-making, and comfort with ambiguous edge cases
• Clear written communication and precise documentation habits