AI/ML Engineer, All Levels

Clera; San Francisco (CA); In loco; USD 120.000 - 200.000; Tempo pieno

Descrizione del lavoro

Clera in San Francisco, CA is seeking an AI/ML engineer to own the intelligence behind failure detection in production agents and verify fixes before deployment.

You'll focus on evaluating when a proposed patch works, turning user signals into evidence, and improving evaluation methods in real-world traces. The role emphasizes reliability, learning from failures, and building trustworthy systems.

Competenze

• Experience shipping products or systems that users rely on.
• Hands-on experience with LLMs in production, including evaluations and embeddings.
• Ability to distinguish meaningful improvements from noise when ground truth is limited.

Mansioni

• Detect subtle agent failures and incorrect responses that appear compliant.
• Turn indirect signals into evidence that an agent has failed.
• Build evaluations to measure detection quality and improve performance.
• Make patch generation trustworthy by reproducing failures and verifying fixes.
• Improve the cost/quality tradeoffs of model-based evaluation.
• Review production traces to identify problems and guide improvements.

Conoscenze

Detect failures

LLMs in production

Evaluation design

Pattern recognition

Descrizione del lavoro

About the Role

Join an AI/ML team focused on making production agents more reliable by detecting failures, evaluating potential fixes, and helping teams build systems they can trust. This role owns the intelligence behind what gets flagged, how confidently it is identified, and whether a proposed fix works.

What You’ll Do

Detect subtle agent failures, including incorrect responses that appear compliant, omissions, and patterns that emerge across many traces.

Turn indirect user signals, such as rephrasing, abandonment, and retries, into evidence that an agent has failed.

Build evaluations that help measure detection quality and improve performance even when labeled ground truth is unavailable.

Make patch generation trustworthy by reproducing failures, verifying fixes, and avoiding low-confidence changes.

Improve the cost and quality tradeoffs of model-based evaluation, including when to use a smaller model or no model.

Review real production traces regularly to identify problems and guide improvements.

What We’re Looking For

• Experience can range from new graduate to approximately 10 years; strong ability to learn and improve is valued.
• A track record of shipping products or systems that people actually use.
• Hands‑on experience with LLMs in production, including evaluations, LLM‑as‑judge approaches, and embeddings.
• Good research judgment and the ability to distinguish meaningful improvements from noise, including when ground truth is limited.
• Experience operating agents in production and learning from their failures is a plus.

Compensation & Benefits

Salary range is $120,000 to $200,000 USD annually. Visa sponsorship is not available.

Location

This is an on‑site role based in San Francisco, United States.