English-Speaking AI Data Evaluation Specialist | Large Language Models

Workster Jobs; Lisboa; On-site; EUR 42,000 - 64,000; Full time

Benefits offered by this job

Private health insurance

Meal allowance

Professional development

Job summary

Workster is partnering with a leading global customer experience company to recruit English-speaking AI Data Evaluation Specialists for an innovative AI project based in Lisbon, Portugal.

Join a team focused on reviewing, evaluating, and improving AI-generated responses to help create safer, more accurate conversational experiences for millions of users worldwide. This office-based role in Lisbon offers onboarding, training, and opportunities to contribute to next-generation AI systems.

Qualifications

• Native-level fluency in English.
• Experience with AI data annotation and LLM evaluation.
• Strong understanding of Generative AI and RLHF basics.
• Excellent analytical thinking and attention to detail.
• Ability to interpret complex guidelines accurately.
• Strong written communication and reading comprehension.

Responsibilities

• Review and evaluate AI-generated responses using annotation guidelines.
• Label and annotate outputs for accuracy and consistency.
• Assess content for quality, relevance, safety, and policy compliance.
• Apply guidelines across diverse content scenarios and edge cases.
• Identify inconsistencies and escalate issues when needed.
• Analyze datasets and AI content to support model improvement.
• Deliver high-quality work meeting productivity and accuracy targets.
• Maintain confidentiality and comply with project standards.

Skills

English fluency

AI data annotation

LLMs evaluation

Analytical thinking

Attention to detail

Reading comprehension

Structured text analysis

Job description

Workster is partnering with a leading global customer experience company to recruit English-speaking AI Data Evaluation Specialists for an innovative artificial intelligence project based in Lisbon, Portugal.

If you’re passionate about AI, language technology, and data quality, this is a unique opportunity to contribute to the development of next-generation Large Language Models (LLMs). You’ll work with cutting-edge AI systems by reviewing, evaluating, and improving AI-generated responses, helping create safer, more accurate, and more reliable conversational experiences for millions of users worldwide.

Your Role

• Review and evaluate AI-generated responses using detailed annotation and quality guidelines

• Label and annotate language model outputs with accuracy and consistency

• Assess AI-generated content for quality, relevance, factual accuracy, safety, and policy compliance

• Apply complex annotation guidelines across a wide variety of scenarios and content types

• Identify inconsistencies, edge cases, and potential model issues, escalating when appropriate

• Analyze structured text, datasets, and AI-generated content to support model improvement

• Deliver high-quality work while consistently meeting productivity, quality, and accuracy targets

• Maintain strict confidentiality and comply with all project standards, security requirements, and internal procedures

• Contribute directly to the ongoing training and improvement of advanced AI systems through detailed evaluations and feedback

• 24/7 rotating schedule (Monday to Sunday) with 2 rotational days off

Your Qualifications

• Native-level fluency in English

• Good command of English (B2 level or above)

• Strong understanding of Generative AI, Large Language Models (LLMs), and AI evaluation methodologies

• Previous experience in AI data annotation, LLM evaluation, Reinforcement Learning from Human Feedback (RLHF), Supervised Fine-Tuning (SFT), Quality Assurance, Content Labeling, Trust & Safety, or similar analytical roles is highly desirable

• Excellent analytical thinking and attention to detail

• Ability to interpret and consistently apply complex written guidelines and policies

• Strong reading comprehension and written communication skills

• Comfortable working with structured text, datasets, and AI-generated content

• Ability to work efficiently in a fast-paced, quality-driven environment while meeting performance targets

• Professional, objective, and capable of handling confidential information with discretion

The Offer

• Office-based position in Lisbon, Portugal

• Comprehensive onboarding and specialized training in AI evaluation processes

• Opportunity to work on one of the world’s most advanced artificial intelligence projects

• International and multicultural working environment

• Career development opportunities within the rapidly growing AI industry

• Private health insurance

• Meal allowance

• Ongoing learning and professional development

• Modern office facilities with a collaborative team culture

• Unique opportunity to shape the future of generative AI and language technologies