AI Engineer: Own Production-Grade, Trustworthy AI

Retool; New York (NY); Hybrid; USD 164,000 - 306,000; Full time

Benefits offered by this job

Medical insurance

Dental coverage

Vision benefits

401(k)

Hybrid work location

Job summary

Retool is hiring an AI Engineer to own model-driven behavior in production across product, infra, and evaluation layers. You’ll shape user experiences and shipping confidence by constraining non-determinism and improving evaluative signals.

You’ll design prompting, routing, and tool-use strategies, build robust evaluation frameworks, and address failures like drift or hallucinations. Strong ownership and independent problem solving are essential.

Qualifications

• 6+ years of professional engineering experience with ownership over complex systems in production.
• Experience owning AI/LLM behavior beyond basic integration, including mitigation of variance and failure modes.
• Comfort reasoning about probabilistic systems and tradeoffs (quality vs. cost, recall vs. precision, speed vs. robustness).
• Experience designing or maintaining evaluation frameworks, golden datasets, regression detection, or human-in-the-loop feedback loops.
• Strong product intuition and ability to operate independently in ambiguous problem spaces.

Responsibilities

• Own model-driven behavior across product surfaces, including quality, safety, variance, and failure modes.
• Design and evolve prompting, routing, and tool-use strategies to bound non-determinism.
• Build and maintain evaluation systems measuring model performance with statistical signals and trends.
• Detect and resolve non-deterministic failures such as hallucinations and drift.
• Collaborate with product and infra teams to decide when probabilistic behavior is ready to ship.

Skills

AI/LLM behavior

Non-determinism

Evaluation frameworks

Probabilistic systems

Product ownership

Tools

TypeScript

Node.js

React

Job description

Retool is hiring an AI Engineer to own model-driven behavior in production across product, infra, and evaluation layers. You’ll shape user experiences and shipping confidence by constraining non-determinism and improving evaluative signals.

You’ll design prompting, routing, and tool-use strategies, build robust evaluation frameworks, and address failures like drift or hallucinations. Strong ownership and independent problem solving are essential.