Production AI/ML Engineer - Reliability & Evaluation (SF)

Wwshemi; San Francisco (CA); On-site; USD 120,000 - 200,000; Full time

Job summary

Wwshemi is seeking an on-site AI/ML Engineer to enhance the reliability of production agents by detecting failures, evaluating fixes, and guiding teams in building trustworthy systems. You will own the intelligence behind flagging decisions and the confidence of identifications.

The role emphasizes hands-on work with LLMs in production, evaluating approaches, and iterating based on real production traces. Salary ranges reflect SF market rates and visa sponsorship is not available.

Qualifications

• Hands-on experience with large language models in production.
• Ability to design and interpret evaluation metrics for model-based systems.
• Experience learning from model failures and improving reliability.

Responsibilities

• Detect subtle agent failures and patterns across traces.
• Turn user signals into evidence of failure and improve detection quality.
• Build evaluations to measure performance even when ground truth is scarce.
• Reproduce failures and verify fixes to ensure patch reliability.
• Assess cost and quality tradeoffs for model-based evaluation.

Skills

LLMs in production

Failure analysis

Evaluation design

Job description

Wwshemi is seeking an on-site AI/ML Engineer to enhance the reliability of production agents by detecting failures, evaluating fixes, and guiding teams in building trustworthy systems. You will own the intelligence behind flagging decisions and the confidence of identifications.

The role emphasizes hands-on work with LLMs in production, evaluating approaches, and iterating based on real production traces. Salary ranges reflect SF market rates and visa sponsorship is not available.