Remote AI Personalization QA Specialist

Turing; New York (NY); On-site; USD 70,000 - 90,000; Full time

Job summary

A leading AI research firm seeks an AI Quality Analyst to evaluate a new personalization feature for Gemini. This role requires proficiency in Russian, strong analytical skills, and the ability to creatively design prompts based on personal experiences. Candidates must be able to work independently and commit to at least 30 hours a week, with flexibility to overlap with PST. This contractor position offers significant involvement in assessing AI model responses and ensuring quality interactions.

Qualifications

• Ability to read and write in Russian with a high degree of competence.
• Willingness to use primary personal Google account and enable personal data sources.
• Full-time availability in your local time zone required.
• Demonstrate ability to evaluate nuanced and ambiguous AI responses.
• Experience in designing creative, multi-turn starting prompts based on personal context.
• Understanding of personalization concepts and ability to identify incorrect personalization.
• Ability to review model responses and spot subtle differences in naturalness.
• Superior ability to write clear, concise, and structured rationales for model rankings.
• Ability to provide constructive feedback and detailed annotations.
• Excellent communication and collaboration skills.
• Self-motivated and able to work independently in a remote setting.
• Desktop/Laptop set up with a good internet connection.

Responsibilities

• Designing and executing multi-turn conversational prompts that require the AI to utilize your personal information.
• Evaluating model responses based on the intent from the starting prompt.
• Analyzing responses for Grounding issues.
• Assessing Integration quality to ensure personal data is woven naturally.
• Evaluating and stack-ranking two model responses side-by-side.
• Writing clear, defensible rationales for your comparisons.
• Extracting and verifying 'Debug Info' from the model.
• Maintaining strict data hygiene by deleting evaluation conversations.

Skills

Russian Proficiency

Exceptional Analytical Thinking

Creative Prompt Engineering

Strong Evaluation Acumen

Meticulous Attention to Detail

Excellent Written Communication

Feedback Capability

Excellent Communication

Independence

Technical Setup

Education

BS/BA degree or equivalent experience

Job description

A leading AI research firm seeks an AI Quality Analyst to evaluate a new personalization feature for Gemini. This role requires proficiency in Russian, strong analytical skills, and the ability to creatively design prompts based on personal experiences. Candidates must be able to work independently and commit to at least 30 hours a week, with flexibility to overlap with PST. This contractor position offers significant involvement in assessing AI model responses and ensuring quality interactions.