AI Output Analysis Specialist (Japanese) | $15/hr Remote

Crossing Hurdles; United States; Remote; USD 60,000 - 80,000; Full time

Job summary

A forward-thinking tech company in the United States is seeking an individual to evaluate AI model responses for personalization quality. Responsibilities include analyzing AI outputs, designing multi-turn prompts, and collaborating with teams to enhance model performance. Ideal candidates will have proficiency in Japanese, experience in AI evaluation, and strong analytical skills. Full-time availability with remote work options is required. This role emphasizes attention to detail and outstanding written communication.

Qualifications

• Excellent reading and writing skills in Japanese.
• Experience in data annotation, AI evaluation, or related role.
• Ability to design creative prompts based on context.

Responsibilities

• Evaluate AI model responses for personalization quality.
• Design and execute multi-turn prompts to test AI capabilities.
• Collaborate with cross-functional teams to improve AI model performance.

Skills

Strong proficiency in Japanese

Analytical thinking

Attention to detail

Written communication

Education

Bachelor’s degree or equivalent experience

Job description

Responsibilities

• Evaluate AI model responses for personalization quality, including grounding, integration, and helpfulness.
• Design and execute multi-turn prompts based on personal context to test AI capabilities.
• Analyze responses for hallucinations, incorrect personalization, and poor inferences.
• Perform side-by-side comparison of model outputs to determine quality and effectiveness.
• Write clear and structured rationales for response evaluations and rankings.
• Extract and verify debug information to ensure proper use of data sources.
• Maintain strict data hygiene and ensure accurate documentation of evaluations.
• Collaborate with cross‑functional teams to improve AI model performance.

Requirements

• Strong proficiency in Japanese with excellent reading and writing skills.
• Experience in data annotation, AI evaluation, content moderation, or a related role.
• Strong analytical thinking and ability to assess nuanced AI responses.
• Ability to design creative, multi-turn prompts based on personal context.
• Understanding of personalization concepts, including identifying incorrect or forced personalization.
• High attention to detail in evaluating subtle differences in model outputs.
• Excellent written communication and structured reasoning skills.
• Ability to work independently in a remote environment.
• Willingness to use a personal Google account for evaluation purposes.
• Full-time availability with at least 4 hours overlap with PST.
• Bachelor’s degree or equivalent experience in a relevant analytical field.