Agentic AI Analyst: Design, Evaluate & Simulate Agents

Invoca; Los Angeles (CA); Remote; USD 94,000 - 134,000; Full time

Benefits offered by this job

Health benefits

Mental wellbeing

Wellness subsidy

Flexible time off

Paid holidays

Paid family leave

Paid medical leave

401(k) plan

Sabbatical

Job summary

Invoca is seeking an Agentic AI Analyst to shape and refine agentic AI apps across product, data science, and customer success. You will design, validate, and operate agent tools, skills, and evaluation harnesses, coordinating with engineering and GTM to translate insights into product priorities.

The role focuses on integration layers, tool schemas, and scalable agent behaviors, with production monitoring and cross-functional collaboration at its core.

Qualifications

• 3+ years in product, analytics, or technical customer-facing roles within SaaS or enterprise software.
• Hands-on experience with agentic AI systems in production environments including multi-step reasoning and tool use.
• Experience designing or integrating tools for AI agents (MCP servers, schemas, API connections).
• Experience with agent design, prompts, and instructions; iterating on results.
• Experience designing and running evals for Agentic systems with golden datasets and rubrics.
• Experience with simulation or synthetic testing approaches for agents.
• Experience with AI orchestration frameworks (e.g., LangChain, LangGraph).
• Ability to work independently on medium-scope initiatives and balance tradeoffs.
• Strong problem-solver with data-driven approach.
• Clear communicator across technical and non-technical stakeholders.
• Collaborative across Engineering, GTM, and Design partners.
• Familiarity with A2A protocols or multi-agent orchestration.
• Experience designing for long-running or stateful agent sessions.

Responsibilities

• Design and Validate Agentic Integrations (MCP Tools) with tool specs and diagnostics.
• Author and Maintain Agent Skills with reusable, versioned libraries.
• Build and Run Agent Evals using gold datasets and rubrics; report results clearly.
• Simulate Agent Behavior before Production with synthetic calls and mock backends.
• Design End-to-End Agent Behavior including planning, state, escalation, and fallback.
• Validate Use Cases with Customers and GTM; translate findings into proposals.
• Monitor Agent Performance in Production; track task completion, tool calls, and feedback.
• Optimize Retrieval and Model Behavior with retrieval and RAG strategies.
• Triage Production Issues and Regressions; document and address root causes.
• Document Agent Designs and Best Practices for reuse across teams.
• Enable Cross-Functional Teams and Customers with collateral and training.
• Design Multi-Agent and Long-Running Workflows and state handling.

Skills

3+ years in product analytics or tech–

Hands-on agentic AI

Tool integration for AI agents

Agent design & architecture

Eval design for Agentic systems

Simulation/testing of agents

AI orchestration frameworks

Independent initiative

Problem decomposition

Strong communicator

Cross-functional collaboration

A2A protocols familiarity

Long-running/stateful agents

Tools

MCP servers

APIs integration

iPaaS tools

Job description

Invoca is seeking an Agentic AI Analyst to shape and refine agentic AI apps across product, data science, and customer success. You will design, validate, and operate agent tools, skills, and evaluation harnesses, coordinating with engineering and GTM to translate insights into product priorities.

The role focuses on integration layers, tool schemas, and scalable agent behaviors, with production monitoring and cross-functional collaboration at its core.