AI Engineer II

Guidewire; Myrtle Point (OR); On-site; USD 140,000 - 200,000; Full time

Job summary

Guidewire is seeking an experienced engineer to build, evaluate, and harden an AI agent system. You will own end-to-end slices of the agent, including prompts, context, data retrieval, tool calls, and handoffs to humans, with a focus on evidence-based improvements.

The role involves building eval loops, golden sets, and trajectory evaluation to gate releases, and shaping the voice path with latency budgets and telephony layers.

Qualifications

• You've shipped an LLM application with real users relying on it, and you understand making output predictable with disciplined evals and metrics.
• You can reason about cost, latency, reliability, and scalability, designing data stores and distributed systems with security and CI/CD in mind.
• You work with coding agents, managing their context and deciding when to intervene vs. let them run, adapting as tooling evolves.
• You participate in shaping the product scope and insurance context to guide MVPs and production workflows.

Responsibilities

• Own end-to-end slices of the agent: prompt and context architecture, data retrieval, tool calls, and human handoffs.
• Build and run evaluation loops, golden sets, and trajectory assessment to gate releases with evidence.
• Improve voice path aspects: take-taking, interruptions, latency budgets, and telephony integration (LiveKit, SIP).
• Listen to carrier calls and recordings to identify root causes and drive fixes beyond logs.

Skills

Building AI apps

Software fundamentals

Coding agents

Shaping the build

Tools

LiveKit

SIP

CI/CD

Job description

The Role:

You will build, evaluate, and harden the agent itself: the orchestration, the context it sees, the tools it can call, and the evals that tell us whether a change made it better or just different. Our rollout philosophy is a dimmer, not a light switch. We widen the agent's autonomy one notch at a time, and every notch has to be earned with evidence. Much of your work is producing that evidence, then acting on what it says.

Job Description

What you'll do Own end-to-end slices of the agent: prompt and context architecture, retrieval over policy and coverage data, tool and API calls into core claims systems, and handoff to a human when the agent should stop. Build and run the eval loop, golden sets, trajectory evaluation, LLM-as-judge, error analysis on real production call traces, and make it the thing that gates releases. Drive containment and resolution rates up without trading away accuracy, and instrument the system well enough that we can tell which of the two moved. Work directly on the voice path: turn-taking, interruption, latency budgets, and the telephony layer underneath (LiveKit, SIP). Sit in on carrier calls and listen to recordings. The best fixes usually come from hearing where the conversation went sideways, not from reading the logs.

What we're looking for

1. Building and deploying AI applications. You've shipped an LLM application that real users depend on, and you know that the hard part is that the output is not predictable. You're fluent in the building blocks LLMs, context engineering, RAG, agentic workflows, and you have real ML and deep learning depth underneath that fluency. Most importantly, you know how to make an unpredictable system behave predictably enough to trust: disciplined evals, error analysis, and statistical measurement rather than vibes and a handful of spot checks.
2. Software engineering fundamentals. You can reason about cost, latency, reliability, and scalability as tradeoffs rather than goals, and you know which ones you're making. You've designed data stores and distributed systems, you write tests, and you take security and privacy seriously while advocating CI/CD process automation. We handle claims data, recorded calls, and PII under carrier and regulatory scrutiny.
3. Using coding agents. You work with coding agents daily and you're good at it: managing their context, deciding when a spec is worth writing and when it isn't, giving them verifiers so they can close their own loops, and knowing when to intervene versus leave them alone. You also keep changing how you work as the tooling changes, rather than settling into a 2025 workflow.
4. Shaping the build. You won't be handed a finished spec. You'll be in the room where we decide what the agent should do next, which means you need product sense and enough insurance context to argue about it. You'll know when to put an MVP in front of a carrier next week and when a claims workflow demands that we slow down and get it right the first time. Underneath all four: you keep learning. This field looks different every six months and we expect to keep changing our minds.

Nice to have

• Real-time or streaming voice systems (ASR, TTS, telephony, SBC/SIPREC)
• Insurance, claims, or another regulated domain where being wrong has a cost attached
• Experience building internal eval or observability tooling other engineers actually adopted.
• Experience in Machine learning principles, or have experience in model experimentations, training and deploying models in Productions.
• Experience in P&C Insurance or exposure to Guidewire products and services.
• Prompt or pipeline optimization frameworks (DSPy, GEPA) applied to a production system