AI Engineer — Production-Grade Model Reliability

Socket.dev; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Socket.dev is seeking an AI Engineer to own model-driven behavior in production, shaping how probabilistic systems perform across features and surfaces. You will work across product, infra, and evaluation layers to ensure reliable, trustworthy AI experiences for users.

You will design prompting, retrieval, routing, and tool-use strategies, build evaluation pipelines, and implement guardrails and degradation paths.

Qualifications

• 6+ years of professional engineering experience with ownership of complex production systems.
• Experience owning AI/LLM behavior beyond basic integration.
• Ability to reason about probabilistic systems and tradeoffs.
• Experience designing or maintaining evaluation frameworks and regression detection.
• Strong product intuition and care for non-deterministic outputs.
• Independent problem solving in ambiguous spaces.

Responsibilities

• Own the behavior of AI-powered features across product surfaces, including quality, safety, variance, and failure modes.
• Design prompting, retrieval, routing, and tool-use strategies for non-deterministic systems that bound downside.
• Build and maintain evaluation systems using statistical signals, distributions, and trends.
• Detect, diagnose, and resolve non-deterministic failures such as hallucinations, partial correctness, or context drift.
• Define guardrails, fallbacks, and degradation paths that keep systems useful when models behave unexpectedly.
• Partner with product and infra teams to decide when probabilistic behavior is ‘good enough’ to ship.
• Influence model selection, model behavior, and tool design to balance quality, cost, latency, and robustness for real user workflows.

Skills

AI/LLM behavior

Evaluation frameworks

Non-determinism handling

Probabilistic reasoning

Independent problem solving

Data-driven iteration

Tools

TypeScript

Node.js

React

Vector databases

Embeddings

Job description

Socket.dev is seeking an AI Engineer to own model-driven behavior in production, shaping how probabilistic systems perform across features and surfaces. You will work across product, infra, and evaluation layers to ensure reliable, trustworthy AI experiences for users.

You will design prompting, retrieval, routing, and tool-use strategies, build evaluation pipelines, and implement guardrails and degradation paths.