AI Reliability Engineer - Probabilistic Systems

Retool Inc.; San Francisco (CA); On-site; USD 164,000 - 306,000; Full time

Benefits offered by this job

Hybrid work location

Medical, dental, vision

401(k)

Job summary

Retool Inc. is seeking an experienced AI engineer to own the behavior of probabilistic features across product surfaces, ensuring reliability, evaluability, and trust in production. You will design prompting strategies, retrieval and routing approaches, and build evaluation systems to measure model performance beyond pass/fail tests.

You’ll collaborate with product and infra teams to ship dependable AI capabilities at scale, shaping runtime behavior with precision and evidence-driven decisions.

Qualifications

• 6+ years of production engineering experience with ownership of complex systems.
• Experience owning AI/LLM behavior beyond basic integration and mitigating variance.
• Ability to reason about probabilistic systems and tradeoffs in real products.
• Proven track record designing evaluation frameworks and regression detection.
• Strong product intuition and ability to set quality standards independently.

Responsibilities

• Own AI-powered feature behavior across product surfaces, including quality, safety, and failure modes.
• Design prompting, retrieval, routing, and tool-use strategies for non-deterministic systems.
• Build and maintain evaluation systems using statistical signals and trends.
• Diagnose non-deterministic failures like hallucinations or drift and implement guardrails.
• Collaborate with product and infra teams to decide when probabilistic behavior is good enough to ship.
• Influence model selection and tooling to balance quality, cost, latency, and robustness.

Skills

AI/LLM ownership

Probabilistic reasoning

Evaluation frameworks

Non-deterministic design

Independence in ambiguity

Product intuition

Stakeholder collaboration

Tools

Vector databases

Retrieval pipelines

Embeddings

Job description

Retool Inc. is seeking an experienced AI engineer to own the behavior of probabilistic features across product surfaces, ensuring reliability, evaluability, and trust in production. You will design prompting strategies, retrieval and routing approaches, and build evaluation systems to measure model performance beyond pass/fail tests.

You’ll collaborate with product and infra teams to ship dependable AI capabilities at scale, shaping runtime behavior with precision and evidence-driven decisions.