Senior AI Agent & LLM Engineer - Quality & Production

Cacheflow; Mountain View (CA); Hybrid; USD 185,000 - 230,000; Full time

Job summary

Otter.ai is seeking a Senior AI Agent & LLM Engineer to build and improve the AI systems behind Otter’s conversational knowledge engine and AI Chat. You will own end-to-end delivery from ambiguous problems to production solutions with a focus on quality and measurable improvements.

You will collaborate across AI, product, infrastructure, and data teams to enhance knowledge extraction, indexing, evaluation datasets, and release gates for scalable AI capabilities.

Qualifications

• 5+ years of AI Agent engineering or related experience.
• Strong backend or distributed-systems engineering skills.
• Hands-on experience building and shipping AI-agent or LLM-powered products.
• Strong focus on AI quality and evaluating nondeterministic systems.
• Can use data, traces, logs, and qualitative examples to identify and resolve complex failures.
• Works effectively across services, technical domains, and organizational boundaries.

Responsibilities

• Build AI-agent experiences for Otter AI Chat to help users reason over conversations, retrieve knowledge, and complete complex tasks.
• Advance Otter’s conversational knowledge engine through better knowledge extraction, annotation, and indexing.
• Develop robust evaluation datasets, automated graders, regression tests, and release gates for AI quality.
• Diagnose failures across models, prompts, retrieval, tools, data pipelines, backend services, and product workflows.
• Build shared agent infrastructure for orchestration, tracing, debugging, retries, sandboxed execution, and observability.
• Improve task completion, correctness, groundedness, reliability, latency, and cost.
• Turn production traces, customer feedback, and usage signals into measurable product and model improvements.
• Collaborate across AI, product, infrastructure, data, security, and application teams to deliver end-to-end capabilities.

Skills

AI agent engineering

Backend engineering

LLM products

AI quality

Data & logs analysis

Cross-functional collaboration

Product judgment

Ownership

Job description

Otter.ai is seeking a Senior AI Agent & LLM Engineer to build and improve the AI systems behind Otter’s conversational knowledge engine and AI Chat. You will own end-to-end delivery from ambiguous problems to production solutions with a focus on quality and measurable improvements.

You will collaborate across AI, product, infrastructure, and data teams to enhance knowledge extraction, indexing, evaluation datasets, and release gates for scalable AI capabilities.