Remote Agentic AI Analyst: Design & Eval of AI Agents

Invoca; New York (NY); Remote; USD 94,000 - 134,000; Full time

Benefits offered by this job

Health benefits

Mental wellbeing support

Wellness subsidy

Flexible time off

Paid holidays

Paid family leave

Job summary

Invoca, the revenue infrastructure company, is hiring an Agentic AI Analyst to shape the emerging agentic AI category within our Product organization. You will collaborate with Product Managers, Engineers, Data Scientists, and Customer Success to identify use cases and operationalize them to improve customer outcomes.

You’ll own the integration layer, author agent skills, build eval harnesses, run simulations, and translate field data into prioritized product work while coordinating with

Qualifications

• 3+ years of experience in product, analytics, or technical customer-facing roles within SaaS or enterprise software.
• Hands-on experience with agentic AI systems in production environments, including multi-step reasoning, tool use, conversational agents and human-in-the-loop agents.
• Experience designing or integrating tools for AI agents, such as building or configuring MCP servers, defining function/tool schemas, or connecting agents to APIs and enterprise systems.
• Experience with agent design and architecture, authoring prompts, skills, or agent instructions and iterating on them based on measured results.
• Experience designing and running evals for Agentic systems, including building golden data sets, defining success criteria, and building scorers.
• Experience with simulation or synthetic testing approaches for agents (e.g., synthetic users, mocked tools, scenario generation).
• Experience with AI orchestration frameworks (e.g., LangChain, LangGraph).
• Demonstrated ability to work independently on medium-scope initiatives and balance tradeoffs across requirements, user value, and implementation effort.
• Strong communicator who can lead and coordinate messaging across technical and non-technical stakeholders.
• Highly collaborative and comfortable working across Engineering, GTM, and Design partners.
• Familiarity with agent-to-agent (A2A) protocols or multi-agent orchestration patterns.
• Experience designing for long-running or stateful agent sessions (context persistence, session resumption, drift over time).

Responsibilities

• Design and validate agentic integrations (MCP tools) and tool schemas.
• Author and maintain reusable agent skills and prompts.
• Build and run agent evaluation frameworks and dashboards.
• Simulate agent behavior with synthetic callers and mock tool backends.
• Define end-to-end agent behavior, planning and state management.
• Validate use cases with customers and GTM; translate findings into scoped proposals.
• Monitor production metrics and drive improvements in agent efficacy.
• Tune retrieval and model behavior to improve agent performance.
• Triage production issues and regressions; document findings and fixes.
• Document agent designs and best practices for reuse across teams.
• Enable cross-functional teams and customers through collateral and training.
• Design multi-agent and long-running workflows, including task handoffs and state management.

Skills

Product analytics experience

Agentic AI systems

AI agent tools integration

Agent design & prompts

Agent evals

Simulation testing

AI orchestration (LangChain)

Independent work

Communication

Collaboration

A2A protocols

Stateful agent sessions

Tools

LangChain

LangGraph

Job description

Invoca, the revenue infrastructure company, is hiring an Agentic AI Analyst to shape the emerging agentic AI category within our Product organization. You will collaborate with Product Managers, Engineers, Data Scientists, and Customer Success to identify use cases and operationalize them to improve customer outcomes.

You’ll own the integration layer, author agent skills, build eval harnesses, run simulations, and translate field data into prioritized product work while coordinating with