Remote Agentic AI Analyst: Shape Real-World AI Tools

Invoca; Austin (TX); On-site; USD 94,000 - 134,000; Full time

Benefits offered by this job

Day-One Health Benefits

Mental Wellbeing

Wellness Subsidy

Flexible Time Off

Paid Holidays

Paid Family Leave

401(k)

Job summary

Invoca is seeking an Agentic AI Analyst to shape and refine the emerging category of agentic AI applications. You will collaborate with Product, Engineers, Data Scientists, and Customer Success to identify and operationalize AI use cases that improve outcomes and advance Invoca's AI vision.

You will design integration layers, author skills, build evals, and run simulations to stress-test agents before customers touch them, growing your product skillset at the intersection of customer needs and

Qualifications

• 3+ years of experience in product, analytics, or technical customer-facing roles within SaaS or enterprise software.
• Hands-on experience with agentic AI systems in production environments, including multi-step reasoning, tool use, conversational agents and human-in-the-loop agents.
• Experience designing or integrating tools for AI agents, such as building or configuring MCP servers, defining function/tool schemas, or connecting agents to APIs and enterprise systems.
• Experience with agent design and architecture, authoring prompts, skills, or agent instructions and iterating on them based on measured results.
• Experience designing and running evals for Agentic systems, including building golden data sets, defining success criteria, and building scorers.
• Experience with simulation or synthetic testing approaches for agents (e.g., synthetic users, mocked tools, scenario generation).
• Experience with AI orchestration frameworks (e.g., LangChain, LangGraph).
• Demonstrated ability to work independently on medium-scope initiatives and balance tradeoffs across requirements, user value, and implementation effort.
• Skilled at breaking down ambiguous problems, gathering data, and proposing thoughtful, data-driven solutions.
• Strong communicator who can lead and coordinate messaging across technical and non-technical stakeholders.
• Highly collaborative and comfortable working across Engineering, GTM, and Design partners.
• Familiarity with agent-to-agent (A2A) protocols or multi-agent orchestration patterns.
• Experience designing for long-running or stateful agent sessions (context persistence, session resumption, drift over time).

Responsibilities

• Design and Validate Agentic Integrations (MCP Tools): Define the tools our agents use to read and act on customer data.
• Author and Maintain Agent Skills: Design, write, and maintain reusable skills: structured instructions, workflows, and domain knowledge that govern how agents approach specific jobs.
• Build and Run Agent Evals: Design evaluation frameworks for agentic behaviors, including golden datasets, task-level success criteria, grading rubrics, and LLM-as-judge pipelines.
• Simulate Agent Behavior Before Production: Build and operate simulation environments to exercise multi-step agent workflows at scale.
• Design End-to-End Agent Behavior: Own how an agent reasons through a task from start to finish, including planning logic and state management across steps.
• Validate Use Cases with Customers and GTM: Work with Sales Engineering, Customer Success, and customers to surface high-value problems and translate findings into scoped proposals.
• Monitor Agent Performance in Production: Track key metrics to assess agent efficacy and prioritize improvements.
• Optimize Retrieval and Model Behavior: Tune retrieval and model components using eval and production data to guide changes.
• Triage Production Issues and Regressions: Capture, reproduce, and document unintended agent behaviors and land fixes.
• Document Agent Designs and Best Practices: Own documentation for agent designs, tool specs, skill libraries, and eval methodologies.
• Enable Cross-Functional Teams and Customers: Develop collateral and training on agentic AI features and represent Product in customer meetings.
• Design Multi-Agent and Long-Running Workflows: Architect how agents hand off tasks and maintain context across long-running interactions.

Skills

Experience in product analytics

Agentic AI systems

MCP tools

Agent design

Agent evals

Simulations

LangChain/LangGraph

Independent work

Analytical thinking

Communication

Collaboration across teams

A2A protocols

Stateful agent sessions

Python

JSON/JSON Schema

APIs and integrations

Marketing tech experience

Job description

Invoca is seeking an Agentic AI Analyst to shape and refine the emerging category of agentic AI applications. You will collaborate with Product, Engineers, Data Scientists, and Customer Success to identify and operationalize AI use cases that improve outcomes and advance Invoca's AI vision.

You will design integration layers, author skills, build evals, and run simulations to stress-test agents before customers touch them, growing your product skillset at the intersection of customer needs and