Remote Agentic AI Analyst Build and Validate AI Agents

Invoca; Denver (CO); On-site; USD 94,000 - 134,000; Full time

Benefits offered by this job

Stock options

Remote-first culture

Health benefits

Paid time off

Job summary

Invoca is seeking an Agentic AI Analyst to shape and refine agentic AI applications across product teams. You will design and validate MCP tool integrations, author reusable agent skills, and run rigorous evals while simulating agent behavior before production.

The role emphasizes collaboration with Engineering, Data Science, and Customer Success to translate field data into product priorities. Key duties include designing end-to-end agent behavior, monitoring production metrics, and driving

Qualifications

• 3+ years in product, analytics, or tech customer-facing roles in SaaS/enterprise software.
• Experience with agentic AI systems in production, including multi-step reasoning and tools.
• Experience designing or integrating tools for AI agents, such as MCP servers and function schemas.
• Experience authoring prompts, skills, or agent instructions and iterating on them based on results.
• Experience designing and running evals for Agentic systems with golden data sets and success criteria.
• Experience with simulation or synthetic testing for agents.
• Familiarity with AI orchestration frameworks (e.g., LangChain, LangGraph).
• Strong independent work and data-driven decision making.
• Strong communicator across technical and non-technical stakeholders.
• Collaborative across Engineering, GTM, and Design partners.
• Familiarity with A2A protocols or multi-agent orchestration.
• Experience designing for long-running/stateful agent sessions.

Responsibilities

• Design and Validate Agentic Integrations (MCP Tools) and tool specifications.
• Author and Maintain Agent Skills and reusable instruction libraries.
• Build and Run Agent Evals with golden datasets and grading rubrics.
• Simulate Agent Behavior Before Production via synthetic callers and mock backends.
• Design End-to-End Agent Behavior including planning, state management, escalation, and fallbacks.
• Validate Use Cases with Customers and GTM and translate findings into proposals.
• Monitor Agent Performance in Production and track key metrics.
• Optimize Retrieval and Model Behavior with retrieval and RAG tuning.
• Triage Production Issues and Regressions and land fixes.
• Document Agent Designs and Best Practices for reuse.
• Enable Cross-Functional Teams and Customers with collateral and training.
• Design Multi-Agent and Long-Running Workflows with context persistence.

Skills

Agentic AI systems

Cross-functional leadership

Problem-solving

Analytics

Communication

Tools

MCP servers

API integration

JSON schemas

LangChain

Job description

Invoca is seeking an Agentic AI Analyst to shape and refine agentic AI applications across product teams. You will design and validate MCP tool integrations, author reusable agent skills, and run rigorous evals while simulating agent behavior before production.

The role emphasizes collaboration with Engineering, Data Science, and Customer Success to translate field data into product priorities. Key duties include designing end-to-end agent behavior, monitoring production metrics, and driving