Agentic AI Analyst: Build Real-World AI Agents

Invoca; Chicago (IL); Remote; USD 94,000 - 134,000; Full time

Benefits offered by this job

Health benefits

Mental wellbeing program

Wellness subsidy

Flexible time off

Paid holidays

Paid family leave

Paid medical leave

401(k) with company match

Equity program

Sabbatical option

Job summary

Invoca, a leading revenue infrastructure company, is seeking an Agentic AI Analyst to shape and refine agentic AI applications across product teams. You’ll collaborate with Product Managers, Engineers, Data Scientists, and Customer Success to identify and operationalize agentic use cases that improve customer outcomes.

Your work will center on integration layers for agents, tooling, skills, evaluations, and simulations to ensure real-world effectiveness before production.

Qualifications

• 3+ years of experience in product, analytics, or technical customer-facing roles within SaaS or enterprise software.
• Hands-on experience with agentic AI systems in production environments, including multi-step reasoning, tool use, conversational agents and human-in-the-loop agents.
• Experience designing or integrating tools for AI agents, such as building or configuring MCP servers, defining function/tool schemas, or connecting agents to APIs and enterprise systems.
• Experience with agent design and architecture, authoring prompts, skills, or agent instructions and iterating on them based on measured results.
• Experience designing and running evals for Agentic systems, including building golden data sets, defining success criteria, and building scorers.
• Experience with simulation or synthetic testing approaches for agents (e.g., synthetic users, mocked tools, scenario generation).
• Experience with AI orchestration frameworks (e.g., LangChain, LangGraph).
• Demonstrated ability to work independently on medium-scope initiatives and balance tradeoffs across requirements, user value, and implementation effort.
• Skilled at breaking down ambiguous problems, gathering data, and proposing thoughtful, data-driven solutions.
• Strong communicator who can lead and coordinate messaging across technical and non-technical stakeholders.
• Highly collaborative and comfortable working across Engineering, GTM, and Design partners.
• Familiarity with agent-to-agent (A2A) protocols or multi-agent orchestration patterns.
• Experience designing for long-running or stateful agent sessions (context persistence, session resumption, drift over time).

Responsibilities

• Design and Validate Agentic Integrations (MCP Tools): Define the tools our agents use to read and act on customer data. Author tool specifications, prototype MCP servers and tool integrations, and validate agent tool usage across scenarios.
• Author and Maintain Agent Skills: Design, write, and maintain reusable skills and knowledge that govern agent behavior.
• Build and Run Agent Evals: Design evaluation frameworks, run evals across models, prompts, tools, and skills, and report results for action.
• Simulate Agent Behavior Before Production: Build simulations to exercise multi-step workflows and identify failure modes.
• Design End-to-End Agent Behavior: Own planning, state management, escalation points, and fallback behavior.
• Validate Use Cases with Customers and GTM: Translate findings into scoped proposals from field engagements.
• Monitor Agent Performance in Production: Track metrics like task completion rates and tool-call success.
• Optimize Retrieval and Model Behavior: Tune retrieval, context management, and model selection based on data.
• Triage Production Issues and Regressions: Capture, reproduce, and document agent issues; drive fixes.
• Document Agent Designs and Best Practices: Create internal docs for patterns and troubleshooting.
• Enable Cross-Functional Teams and Customers: Create collateral and training for agentic features and demonstrations.
• Design Multi-Agent and Long-Running Workflows: Architect task handoffs and context persistence across sessions.

Skills

Product analytics

Agentic AI systems

Tool integration

Eval design

Simulation testing

AI orchestration

Strong communicator

Cross-functional work

Tools

MCP tools

LangChain

API integration

Job description

Invoca, a leading revenue infrastructure company, is seeking an Agentic AI Analyst to shape and refine agentic AI applications across product teams. You’ll collaborate with Product Managers, Engineers, Data Scientists, and Customer Success to identify and operationalize agentic use cases that improve customer outcomes.

Your work will center on integration layers for agents, tooling, skills, evaluations, and simulations to ensure real-world effectiveness before production.