Principal AI Engineer, Vertical SaaS

Monarch AI

San Francisco (CA)

On-site

USD 210,000 - 360,000

Full time

4 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Excellent healthcare
Life insurance
401k+ benefits
PTO

Job summary

Monarch AI is hiring a Principal AI Engineer to lead the development of enterprise-grade AI agents and multi-agent systems in a rapidly growing SaaS environment. You will build durable, testable, production-ready agent architectures, focusing on reliability, observability, and safe deployment in enterprise settings.

The role emphasizes ship-ready design, tool-calling reliability, and managing memory, prompts, and state across complex tasks, with a high-reward, high-effort work culture.

Qualifications

  • Proven experience building enterprise-grade AI agents for production.
  • Experience shipping 0-1 multi-tenant SaaS products.
  • Deep expertise in system prompts, multi-step agent design, and memory management.

Skills

Enterprise AI agents
Multi-tenant SaaS
Agent architecture
Tool-calling reliability
Memory architecture
Evaluation and testing
Observability and logging
Guardrails and safety
System design and state management
Production-grade deployment

Job description

Compensation: $200+ base (DOE) + performance-based incentive + options + excellent healthcare and life insurance, 401k+ benefits, PTO

Monarch AI is hiring a Principal-level AI Engineer to join our rapidly growing agentic AI vertical SaaS company, where we are uniquely positioned to dominate a number of vertical sectors serving building products & materials manufacturers. We're looking for Engineers who are exceptional builders of AI agents, who have built a 0-1 multi-tenant SaaS product before, and who ideally has enterprise, experience shipping production-grade AI agents at scale. A data science background is also a major plus.

This role is for someone with proven experience building enterprise-grade, production-ready AI Agents and multi-agent LLM systems, with deep expertise in how AI agents behave in enterprise environments — and who has personally built and deployed multiple AI agents and/or LLM-based systems into production.

We're looking for an individual contributor who wakes up and wants to build innovative AI Agents and valuable Agentic software each and every day. This is a work-hard/high-reward position where exceptional work is rewarded.

We are looking for an engineer who has deep experience in:

  • Building enterprise production-grade AI agents
  • Has shipped multi-tenant software as a major individual contributor
  • system prompt architecture, and multi-step agent design
  • tool-calling reliability, retries, fallback logic, and failure handling under real-world conditions
  • eval frameworks, regression testing, benchmarking, and how to measure agent quality over time
  • agent memory architecture, including short-term vs. long-term memory, retrieval-based memory, state persistence, memory relevance filtering, and the tradeoffs between useful memory, noisy memory, and dangerous memory corruption
  • agent state management, session continuity, workflow state, and how to preserve context across complex multi-step tasks without creating brittle behavior
  • hallucination prevention, edge-case handling, and designing systems that remain trustworthy when models are uncertain, wrong, or inconsistent
  • structured outputs, schema enforcement, validation layers, and techniques to ensure model responses can safely drive downstream actions
  • retrieval design, context management, chunking strategy, ranking, context injection, and how to maximize useful recall without overwhelming the model
  • model quality, latency, and cost tradeoffs, including when to use stronger models, weaker models, routing layers, and staged reasoning patterns
  • observability, tracing, logging, and how to inspect agent decisions, tool calls, and failure points in production
  • guardrails, approval flows, permissioning, and human-in-the-loop controls for high-risk or irreversible actions
  • orchestration of multi-agent systems, including delegation, task decomposition, coordination logic, and avoiding unnecessary agent complexity
  • context window limitations, token budgeting, and designing around model memory constraints in practical production environments
  • reliability engineering for LLM systems, including idempotency, retry safety, partial failure recovery, and graceful degradation
  • model drift, prompt drift, and how to maintain system performance as models, workflows, and user behavior evolve over time
  • real-world production tradeoffs between agent autonomy and deterministic software control

We want the kind of AI engineer who knows how to turn LLM capability into systems that are durable, testable, production-grade, and truly enterprise-ready.

  • Ideal background: Proven experience building and deploying multiple production-grade AI agents and/or LLM systems that have operated in real-world environments, not just prototypes or internal demos
  • Strong hands-on experience with enterprise-ready AI systems, including reliability, scalability, maintainability, observability, and safe deployment practices
  • able to think deeply about agent architecture, workflow design, memory, orchestration, tool usage, and state management — not just prompt writing
  • understands where and why agents break in production, including hallucinations, tool failures, context loss, memory corruption, brittle workflows, and edge-case behavior — and knows how to engineer around those realities
  • experienced building durable AI systems with proper evals, regression testing, guardrails, structured outputs, validation layers, logging, tracing, and iteration loops
  • understands how to balance model quality, latency, and cost while still delivering strong real-world performance at scale
  • able to design systems that are not only intelligent, but also testable, debuggable, auditable, and trustworthy in an enterprise setting
  • comfortable building from zero in ambiguous environments, while still bringing structure, engineering rigor, and clear technical judgment
  • has high standards, low ego, and fast execution speed — someone who cares about getting the system right without becoming overly academic or slow
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Agentic AI Engineer
Agentic AI Engineer

Tactical Edge • Washington

On-site
USD 150,000 - 210,000
AI Engineer - Forward Deployed
AI Engineer - Forward Deployed

Moring, inc. • Atlanta (GA)

On-site
USD 120,000 - 160,000
Agentic AI Developer
Agentic AI Developer

Xpetize • United States

Remote
USD 120,000 - 180,000
Principal AI Engineer
Principal AI Engineer

GSquared Group • Seattle (WA)

On-site
USD 180,000 - 240,000
AI Engineer
AI Engineer

Moring, inc. • Alpharetta (GA)

On-site
USD 60,000 - 80,000
Staff Applied Scientist
Staff Applied Scientist

Robots & Pencils • California

On-site
USD 125,000 - 173,000
Founding AI Engineer
Founding AI Engineer

Rise Technical Recruitment Ltd • San Francisco (CA)

On-site
USD 120,000 - 200,000
$120,000 - $200,000 base salary
0.5 - 2% equity
Flexible PTO
Senior AI Engineer (LLMs / Agents)
Senior AI Engineer (LLMs / Agents)

Rise Technical Recruitment Ltd • United States

Hybrid
USD 180,000 - 250,000
Equity
Healthcare
PTO
Agentic AI Engineer
Agentic AI Engineer

Trilagen • Bethesda (MD)

Hybrid
USD 120,000 - 160,000
401K
Health Insurance
Dental Insurance
+2
Senior AI Engineer, Release Engineering
Senior AI Engineer, Release Engineering

Intercontinental Exchange • Atlanta (GA)

On-site
USD 140,000 - 200,000