Remote Fintech AI Evaluation Lead

elly; United States; Remote; USD 120,000 - 140,000; Full time

Job summary

The Mission is seeking an AI Evaluation Lead for a fully remote role open to candidates based in the United States. You will own how we measure the quality of AI-generated financial guidance, using an existing test-case library and scoring infrastructure.

This is not a monitor-and-report job; it requires judgment on model behavior and data signals to ensure safe, accurate outputs. You will work with the AI/ML team, founders, and subject matter experts to define evaluation criteria, interpret

Qualifications

• Experience evaluating AI/ML system quality with high-stakes outputs.
• Ability to reason about data signals and what they imply for model behavior.
• Comfort making judgment calls in ambiguous situations rather than waiting for obvious answers.

Responsibilities

• Define and validate evaluation sets to ensure domain coverage.
• Analyze scoring results to identify patterns and high-frequency case types.
• Assess whether current measures detect correct failure modes and adjust as needed.
• Review flagged cases and decide on next steps using data and domain knowledge.
• Calibrate when human review is triggered and ensure thresholds stay aligned as the platform scales.
• Collaborate with SMEs to incorporate domain insights into evaluation design.
• Maintain coverage with new domains and model changes before release.
• Translate findings into actionable recommendations for the AI/ML team.

Skills

AI quality

Data analysis

Domain judgment

Tools

Evaluation tooling

Job description

The Mission is seeking an AI Evaluation Lead for a fully remote role open to candidates based in the United States. You will own how we measure the quality of AI-generated financial guidance, using an existing test-case library and scoring infrastructure.

This is not a monitor-and-report job; it requires judgment on model behavior and data signals to ensure safe, accurate outputs. You will work with the AI/ML team, founders, and subject matter experts to define evaluation criteria, interpret