AI Quality Analyst: Prompt Design & Personalization Evaluation

Hire Feed; Pakistan; On-site; PKR 800,000 - 1,100,000; Full time

Job summary

Hire Feed is seeking an AI Quality Analyst to join a client's team on a full-time basis. The role focuses on evaluating a new personalization feature for an AI model, ensuring effective integration of past conversations, emails, searches and activity to yield relevant responses.

The position blends creative prompt design with rigorous analysis of AI outputs, emphasizing grounding, relevance and helpfulness, with detailed feedback to boost personalization performance.

Qualifications

• Proficient in English reading and writing.
• Experience in prompt engineering or AI test design.
• Ability to analyze nuanced AI responses with analytical rigor.
• Familiar with personalization concepts and evaluation metrics in AI systems.
• Attention to detail to identify subtle discrepancies in model outputs.
• Willingness to use your personal Google account and enable personal data sources for assessment.
• Full-time availability in your local time zone to support a global team.

Responsibilities

• Evaluate the quality of personalized responses from an AI model using your own personal experiences as context.
• Design creative, multi-turn prompts that test the model’s ability to integrate personal data sources accurately.
• Assess model responses for grounding, integration, and helpfulness, identifying incorrect personalization or forced connections.
• Review side-by-side comparisons of AI responses to detect subtle differences in relevance and accuracy.
• Provide detailed feedback on model behavior to improve personalization performance.

Skills

English proficiency

Prompt engineering

Analytical evaluation

Personalization concepts

Evaluation metrics in AI

Attention to detail

Google account data sources

Local time zone availability

Job description

Hire Feed is seeking an AI Quality Analyst to join a client's team on a full-time basis. The role focuses on evaluating a new personalization feature for an AI model, ensuring effective integration of past conversations, emails, searches and activity to yield relevant responses.

The position blends creative prompt design with rigorous analysis of AI outputs, emphasizing grounding, relevance and helpfulness, with detailed feedback to boost personalization performance.