AI/ML Engineer — Production Reliability & Evaluation (SF)

Clera; San Francisco (CA); On-site; USD 120,000 - 200,000; Full time

Job summary

Clera in San Francisco, CA is seeking an AI/ML engineer to own the intelligence behind failure detection in production agents and verify fixes before deployment.

You'll focus on evaluating when a proposed patch works, turning user signals into evidence, and improving evaluation methods in real-world traces. The role emphasizes reliability, learning from failures, and building trustworthy systems.

Qualifications

• Experience shipping products or systems that users rely on.
• Hands-on experience with LLMs in production, including evaluations and embeddings.
• Ability to distinguish meaningful improvements from noise when ground truth is limited.

Responsibilities

• Detect subtle agent failures and incorrect responses that appear compliant.
• Turn indirect signals into evidence that an agent has failed.
• Build evaluations to measure detection quality and improve performance.
• Make patch generation trustworthy by reproducing failures and verifying fixes.
• Improve the cost/quality tradeoffs of model-based evaluation.
• Review production traces to identify problems and guide improvements.

Skills

Detect failures

LLMs in production

Evaluation design

Pattern recognition

Job description

Clera in San Francisco, CA is seeking an AI/ML engineer to own the intelligence behind failure detection in production agents and verify fixes before deployment.

You'll focus on evaluating when a proposed patch works, turning user signals into evidence, and improving evaluation methods in real-world traces. The role emphasizes reliability, learning from failures, and building trustworthy systems.