AI Engineer: Build Reliable Non-Deterministic AI in Production

Retool, Inc.; New York (NY); Hybrid; USD 164,000 - 306,000; Full time

Benefits offered by this job

Health insurance

401(k) plan

Hybrid work location

Job summary

Retool, Inc. is seeking an AI Engineer to own model-driven behavior in production across product surfaces. You will shape evaluation, metrics, and reliability for probabilistic systems, working with product and infrastructure teams to ensure safe, high‑quality AI features.

Strong experience in TypeScript/Node.js/React is expected, with a focus on robust governance of AI behavior. You will design prompts, tooling strategies, and evaluation pipelines while balancing quality, cost, and latency in a

Qualifications

• 6+ years of professional engineering experience with ownership over complex systems in production.
• Experience owning AI/LLM behavior beyond basic integration, including mitigation of variance and failure modes.
• Comfort reasoning about probabilistic systems and tradeoffs (quality vs. cost, recall vs. precision, speed vs. robustness).
• Experience designing or maintaining evaluation frameworks, golden datasets, regression detection, or human-in-the-loop feedback loops.
• Strong product intuition about what “good” looks like when outputs are non-deterministic.
• Ability to operate independently in ambiguous problem spaces and set quality standards for others.

Responsibilities

• Own model-driven behavior across product surfaces, including quality, safety, variance, and failure modes.
• Design and evolve prompting, retrieval, routing, and tool-use strategies that embrace non-determinism while bounding its downside.
• Build and maintain evaluation systems that measure model performance using statistical signals, distributions, and trends.
• Detect, diagnose, and resolve non-deterministic failures such as hallucinations or context sensitivity.
• Define and implement guardrails, fallbacks, and degradation paths for unreliable model behavior.
• Partner with product and infra teams to decide when probabilistic behavior is good enough to ship.
• Influence model selection, behavior, and tool design to balance quality, cost, latency, and robustness.

Skills

AI/LLM ownership

Non-deterministic systems

Evaluation frameworks

Probabilistic reasoning

Product intuition

Independent work

Opinions, but open to data

Tools

TypeScript

Node.js

React

Job description

Retool, Inc. is seeking an AI Engineer to own model-driven behavior in production across product surfaces. You will shape evaluation, metrics, and reliability for probabilistic systems, working with product and infrastructure teams to ensure safe, high‑quality AI features.

Strong experience in TypeScript/Node.js/React is expected, with a focus on robust governance of AI behavior. You will design prompts, tooling strategies, and evaluation pipelines while balancing quality, cost, and latency in a