AI Behavior & Personalization Analyst (Japanese) | $15/hr Remote

Crossing Hurdles; United States; On-site; USD 60,000 - 80,000; Full time

Job summary

A tech company specializing in AI evaluation is seeking a full-time evaluator. Responsibilities include assessing AI model responses for quality and effectiveness, designing creative multi-turn prompts, and collaborating with cross-functional teams to enhance performance. The ideal candidate will have strong proficiency in Japanese, excellent analytical and communication skills, and must be available for at least 4 hours overlap with PST. This role can be performed remotely and requires attention to detail in evaluations.

Qualifications

• Strong proficiency in Japanese with excellent reading and writing skills.
• Experience in data annotation, AI evaluation, or content moderation.
• Understanding of personalization concepts and AI response evaluation.

Responsibilities

• Evaluate AI model responses for personalization quality.
• Design multi-turn prompts based on personal context.
• Collaborate with teams to improve AI model performance.

Skills

Strong proficiency in Japanese

Analytical thinking

Content moderation

Attention to detail

Written communication

Education

Bachelor’s degree or equivalent experience in a relevant analytical field

Job description

Responsibilities

• Evaluate AI model responses for personalization quality, including grounding, integration, and helpfulness.
• Design and execute multi-turn prompts based on personal context to test AI capabilities.
• Analyze responses for hallucinations, incorrect personalization, and poor inferences.
• Perform side-by-side comparison of model outputs to determine quality and effectiveness.
• Write clear and structured rationales for response evaluations and rankings.
• Extract and verify debug information to ensure proper use of data sources.
• Maintain strict data hygiene and ensure accurate documentation of evaluations.
• Collaborate with cross‑functional teams to improve AI model performance.

Requirements

• Strong proficiency in Japanese with excellent reading and writing skills.
• Experience in data annotation, AI evaluation, content moderation, or a related role.
• Strong analytical thinking and ability to assess nuanced AI responses.
• Ability to design creative, multi-turn prompts based on personal context.
• Understanding of personalization concepts, including identifying incorrect or forced personalization.
• High attention to detail in evaluating subtle differences in model outputs.
• Excellent written communication and structured reasoning skills.
• Ability to work independently in a remote environment.
• Willingness to use a personal Google account for evaluation purposes.
• Full-time availability with at least 4 hours overlap with PST.
• Bachelor’s degree or equivalent experience in a relevant analytical field.