AI/ML Engineer, All Levels

Wwshemi; San Francisco (CA); On-site; USD 120,000 - 200,000; Full time

Job summary

Wwshemi is seeking an on-site AI/ML Engineer to enhance the reliability of production agents by detecting failures, evaluating fixes, and guiding teams in building trustworthy systems. You will own the intelligence behind flagging decisions and the confidence of identifications.

The role emphasizes hands-on work with LLMs in production, evaluating approaches, and iterating based on real production traces. Salary ranges reflect SF market rates and visa sponsorship is not available.

Qualifications

• Hands-on experience with large language models in production.
• Ability to design and interpret evaluation metrics for model-based systems.
• Experience learning from model failures and improving reliability.

Responsibilities

• Detect subtle agent failures and patterns across traces.
• Turn user signals into evidence of failure and improve detection quality.
• Build evaluations to measure performance even when ground truth is scarce.
• Reproduce failures and verify fixes to ensure patch reliability.
• Assess cost and quality tradeoffs for model-based evaluation.

Skills

LLMs in production

Failure analysis

Evaluation design

Job description

About the Role

Join an AI/ML team focused on making production agents more reliable by detecting failures, evaluating potential fixes, and helping teams build systems they can trust. This role owns the intelligence behind what gets flagged, how confidently it is identified, and whether a proposed fix works.

What You'll Do

• Detect subtle agent failures, including incorrect responses that appear compliant, omissions, and patterns that emerge across many traces.
• Turn indirect user signals, such as rephrasing, abandonment, and retries, into evidence that an agent has failed.
• Build evaluations that help measure detection quality and improve performance even when labeled ground truth is unavailable.
• Make patch generation trustworthy by reproducing failures, verifying fixes, and avoiding low-confidence changes.
• Improve the cost and quality tradeoffs of model-based evaluation, including when to use a smaller model or no model.
• Review real production traces regularly to identify problems and guide improvements.

What We're Looking For

• Experience can range from new graduate to approximately 10 years; strong ability to learn and improve is valued.
• A track record of shipping products or systems that people actually use.
• Hands-on experience with LLMs in production, including evaluations, LLM-as-judge approaches, and embeddings.
• Good research judgment and the ability to distinguish meaningful improvements from noise, including when ground truth is limited.
• Experience operating agents in production and learning from their failures is a plus.

Compensation & Benefits

Salary range is $120,000 to $200,000 USD annually. Visa sponsorship is not available.

Location

This is an on-site role based in San Francisco, United States.