Senior AI Engineer — Own Production Model Behavior

Retool; San Francisco (CA); Hybrid; USD 164,000 - 306,000; Full time

Benefits offered by this job

Medical, dental, vision

401(k)

Job summary

Retool is seeking an AI Engineer to own model-driven behavior in production across product surfaces, shaping prompts, retrieval, routing, and tool usage to manage non-determinism. You will build evaluation systems and guardrails to ensure reliable, trustworthy AI features, collaborating with product and infra teams in a fast-paced environment.

You’ll work with TypeScript/Node.js/React across the stack, owning AI behavior from design to deployment, and driving non-deterministic systems toward

Qualifications

• 6+ years of professional engineering experience with ownership over complex systems in production.
• Demonstrated experience owning AI/LLM behavior beyond basic integration, including mitigation of variance and failure modes.
• Experience designing or maintaining evaluation frameworks, golden datasets, regression detection, or human-in-the-loop feedback loops.

Responsibilities

• Own model-driven behavior in production across multiple product surfaces.
• Design prompting, retrieval, routing, and tool-use strategies that handle non-determinism.
• Build and maintain evaluation systems measuring model performance using statistical signals and trends.
• Detect and resolve non-deterministic failures such as hallucinations or context sensitivity.
• Define guardrails, fallbacks, and degradation paths for unreliable model behavior.

Skills

AI/LLM behavior

Non-deterministic systems

Evaluation frameworks

Product intuition

Independent work

Mentoring engineers

Tools

RAG

Vector databases

Retrieval pipelines

Job description

Retool is seeking an AI Engineer to own model-driven behavior in production across product surfaces, shaping prompts, retrieval, routing, and tool usage to manage non-determinism. You will build evaluation systems and guardrails to ensure reliable, trustworthy AI features, collaborating with product and infra teams in a fast-paced environment.

You’ll work with TypeScript/Node.js/React across the stack, owning AI behavior from design to deployment, and driving non-deterministic systems toward