Remote Agentic AI Analyst: Design & Validate AI Agents

Invoca; Austin (TX); Remote; USD 94,000 - 134,000; Full time

Benefits offered by this job

Day-One Health Benefits

Mental Wellbeing Resources

Wellness Subsidy

Flexible Time Off

Job summary

Invoca is seeking an Agentic AI Analyst to shape and optimize agentic AI applications across product, data, and customer success teams. You will design integrations, author reusable agent skills, and build evaluation frameworks, while collaborating with engineering, GTM, and design to deliver measurable outcomes.

You will validate tool integrations, run simulations, and monitor system performance, driving improvements from field engagements through data-driven priorities.

Qualifications

• 3+ years in product, analytics, or technical customer-facing roles in SaaS or enterprise software.
• Hands-on experience with agentic AI systems in production environments.
• Experience designing or integrating tools for AI agents, such as MCP servers, tool schemas, or API integrations.
• Experience with agent design, prompts, skills, or agent instructions and iterating on them based on results.
• Experience designing and running evals for Agentic systems, including golden datasets and success criteria.
• Familiarity with agent-to-agent (A2A) protocols or multi-agent orchestration patterns.

Responsibilities

• Design and Validate Agentic Integrations (MCP Tools): define tool specs, prototype MCP servers, and validate tool usage across scenarios.
• Author and Maintain Agent Skills: write and manage reusable skills and workflows.
• Build and Run Agent Evals: design evaluation frameworks and run splits to measure performance.
• Simulate Agent Behavior Before Production: build simulations to stress-test multi-step workflows.
• Design End-to-End Agent Behavior: plan task reasoning, state management, escalation, and fallbacks.
• Validate Use Cases with Customers and GTM: surface problems and propose experiment-ready plans.
• Monitor Agent Performance in Production: track task completion, tool-call success, and feedback.
• Optimize Retrieval and Model Behavior: tune retrieval, context, and model selection.

Skills

Product & analytics

Technical customer-facing

Agentic AI systems

Tool integration design

Evaluation & testing

Cross-functional collaboration

Tools

MCP servers

LangChain

Job description

Invoca is seeking an Agentic AI Analyst to shape and optimize agentic AI applications across product, data, and customer success teams. You will design integrations, author reusable agent skills, and build evaluation frameworks, while collaborating with engineering, GTM, and design to deliver measurable outcomes.

You will validate tool integrations, run simulations, and monitor system performance, driving improvements from field engagements through data-driven priorities.