Remote Evaluation Engineer — AI Quality & Safety

Momento USA LLC; United States; Hybrid; USD 90,000 - 170,000; Full time

Job summary

Momento USA LLC is seeking an Evaluation Engineer for a long-term contract, offering remote work with flexibility to align to PST timezone. The role centers on ensuring AI systems meet a defined quality bar before launch and through production, with ongoing monitoring.

You will build eval harnesses, create gold datasets with clinical reviewers, and set pass thresholds. Expect monthly reporting of pass rates and incidents, plus production drift monitoring in collaboration with the pipeline team.

Qualifications

• 4 years in ML/LLM evaluation, QA engineering for AI systems, or applied research engineering.
• Hands-on with eval frameworks and LLM-as-judge patterns and their failure modes, statistical rigor on small samples.
• Independent spine: the team shipping a thing does not set its own pass bar.

Responsibilities

• Build the central eval harness and templates every pod uses; hub-and-spoke central standards, pod-written tests
• Create golden datasets with business and clinical reviewers, including safety and medical-accuracy suites
• Set and defend pass thresholds; audit pod evals; report eval pass rates and incidents monthly
• Stand up production monitoring for drift and regression with the pipeline engineer

Skills

ML/LLM evaluation

QA engineering

Statistical analysis

Eval frameworks

Job description

Momento USA LLC is seeking an Evaluation Engineer for a long-term contract, offering remote work with flexibility to align to PST timezone. The role centers on ensuring AI systems meet a defined quality bar before launch and through production, with ongoing monitoring.

You will build eval harnesses, create gold datasets with clinical reviewers, and set pass thresholds. Expect monthly reporting of pass rates and incidents, plus production drift monitoring in collaboration with the pipeline team.