AI Personal Workflow Evaluator

Obsidian; New York (NY); Remote; USD 34,000 - 55,000; Full time

Job summary

Obsidian is seeking experienced evaluators to test AI systems on complex personal workflows across health, travel, planning, home services, and career search. You will create realistic prompts, execute tasks with screen recording, and use your own plugins to complete actions, providing detailed analyses of AI performance.

Ideal candidates are heavy users of LLM products with 6+ months of activity and a willingness to sign a data-sharing consent.

Qualifications

• US-based only.
• Strong MCP experience and plug-in / connector usage.
• Experience using LLM plugins/connectors such as Google Drive, Expedia, Notion, and similar tools, multiple times a week.
• Heavy personal usage of LLM products.
• An active, rich LLM account with regular usage and approximately 6+ months of history.
• Willingness to sign a data-share consent form via DocuSign.
• Experience using AI for multi-step planning, research, decision-making, or personal workflows.
• Strong written judgment and attention to detail.
• Ability to explain what makes an AI output good, bad, incomplete, unsafe, or unrealistic.
• Experience writing and evaluating against rubrics.
• Extensive rubric experience is valuable (100+ hours).

Responsibilities

• Create realistic prompts for complex personal-life tasks.
• Execute tasks and record your screen (required).
• Use personal plugins/connectors while completing actions.
• Write clear explanations of AI successes and failures.
• Judge practicality, personalization and reasonableness of AI outputs.
• Identify where models miss context or misuse tools.
• Create and apply rubrics to assess model performance.
• Evaluate rubric-driven results during tasks.

Skills

MCP experience

Plugin usage

Tools

Google Drive

Expedia

Notion

Job description

Obsidian is seeking experienced evaluators to test AI systems on complex personal workflows across health, travel, planning, home services, and career search. You will create realistic prompts, execute tasks with screen recording, and use your own plugins to complete actions, providing detailed analyses of AI performance.

Ideal candidates are heavy users of LLM products with 6+ months of activity and a willingness to sign a data-sharing consent.