Conversational Designer

Catapult Solutions Group

United States

Remote

USD 57,000 - 82,000

Full time

28 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Catapult Solutions Group is seeking a Conversation Designer for an all-remote contract role (US, Pacific Time hours) focused on AI chatbot evaluations. You will build datasets, review responses, and score quality against rubrics, collaborating with product, engineering and data science teams.

The ideal candidate has 3–5 years in conversation design, strong analytical skills, and comfort using AI tools. This role offers a 12‑month contract with remote work from anywhere in the US.

Qualifications

  • 3–5 years of experience in conversation design or related AI evaluation
  • Experience evaluating chatbot responses for accuracy and relevance
  • Familiarity with structured content, rubrics and QA processes
  • Strong analytical and logical reasoning skills
  • Excellent written communication and attention to detail

Responsibilities

  • Create large datasets of realistic customer questions with verified correct answers
  • Develop question-and-answer pairs to evaluate chatbot performance
  • Review AI-generated responses for accuracy, relevance and quality
  • Compare chatbot answers against approved source content and documentation
  • Score responses using evaluation rubrics and design principles
  • Identify incorrect answers and recurring error patterns; report findings to Engineering
  • Calculate accuracy percentages and monitor performance trends
  • Provide actionable feedback to Engineering and product partners
  • Use AI tools to streamline evaluation workflows
  • Build simple tools or automations with AI-assisted tools when helpful

Skills

Conversation design
AI/LLM evaluation
Analytical thinking
Attention to detail
Pacific Time availability

Education

Bachelor's degree preferred

Tools

Claude Code
ChatGPT

Job description

Conversation Designer - AI Chatbot Evaluations

Location: 100% Remote (US) - must work Pacific Time business hours

Role Type: Contract, 12 months (W-2)

About Our Client

Our client is a global financial technology company, headquartered in the San Francisco Bay Area. Its tax, accounting, personal finance and marketing products serve roughly 100 million consumers and small businesses worldwide, and its small-business accounting platform is one of the most widely used in the US. Over the past few years, the company has been transforming into an AI-driven platform, building generative-AI assistants and agents into its products. Its mission is to power prosperity around the world, and it has a reputation for customer obsession, innovation and a strong engineering and design culture.

Job Description

Our client's small-business platform includes an AI chatbot, built on a leading large language model and extended with company-specific skills and agents, that helps business owners get answers and complete tasks in the product. As more customers rely on it, making sure every answer is accurate, relevant and on-brand is critical. That's why the team is adding three Conversation Designers focused on AI chatbot evaluation and quality.

In this role, you'll build the question-and-answer datasets used to test the chatbot, review and score its responses against approved source content and quality rubrics, and track accuracy over time. When quality drops in an area, you'll dig in, find the pattern behind the errors and give Engineering clear, actionable feedback. You'll work closely with conversation designers, product managers, engineers and data scientists on the conversational AI team.

This is not a visual UX/UI design role and not a software engineering role, and no Python or coding is required. It's ideal for an analytical, detail-oriented person who loves AI, enjoys figuring out why a chatbot got something wrong, and uses AI tools (such as Claude Code or ChatGPT) to work faster and smarter.

Duties and Responsibilities
  • Create large datasets of realistic customer questions with verified correct answers.
  • Develop question-and-answer pairs used to evaluate chatbot performance.
  • Review AI-generated responses for accuracy, relevance and overall quality.
  • Compare chatbot answers against approved source content and documentation.
  • Score responses using established evaluation rubrics and conversation design principles.
  • Identify incorrect answers, inconsistencies and recurring error patterns.
  • Calculate accuracy percentages and monitor performance trends over time.
  • Investigate areas where chatbot quality is declining and pinpoint likely causes.
  • Provide clear, actionable feedback to Engineering and product partners.
  • Use AI tools to streamline evaluation and quality-assurance workflows.
  • Build simple productivity tools or automations with AI-assisted tools (e.g., Claude Code) when helpful.
Required Experience/Skills
  • 3-5 years of experience in conversation design, chatbot design, conversational AI, AI/LLM evaluation or closely related work (strong candidates with less experience will be considered).
  • Hands-on experience evaluating chatbot or AI responses for accuracy and relevance.
  • Experience working with structured content, evaluation criteria or quality rubrics.
  • Comfortable reviewing large datasets and identifying patterns and trends.
  • Ability to calculate basic metrics (percentages, accuracy rates) and analyze trends.
  • Strong analytical and logical reasoning skills.
  • Ability to understand and consistently apply existing design principles and guidelines.
  • Exceptional attention to detail.
  • Comfortable learning and using AI-powered tools in daily work.
  • Clear written communication for documenting findings and giving feedback.
  • Able to work Pacific Time business hours.
Nice-to-Haves
  • Experience with Claude Code or similar AI-assisted tools.
  • Experience with LLM evaluations, AI data labeling/annotation or response quality assessment.
  • Familiarity with generative AI workflows and prompt writing.
  • Experience at a SaaS or technology company.
  • Knowledge of small-business accounting software or bookkeeping concepts.
  • Experience building simple tools or automations using AI.
  • Background as a technical writer or knowledge base specialist who has worked on chatbot content and AI evaluation.
Education

Bachelor's degree preferred (e.g., HCI, Interaction Design, Linguistics, Psychology, Computer Science or a related field), or equivalent practical experience.

Pay & Benefits Summary
  • Pay rate: $50.35/hr (W-2 contract).
  • 12-month contract.
  • 100% remote - work from anywhere in the US on Pacific Time hours.
  • Hands-on work with cutting-edge generative AI on a product used by millions of small businesses.

Love AI and have a sharp eye for when a chatbot gets it wrong? Work from the comfort of your home and help shape the quality of an AI assistant used by millions.

Conversation Designer | Conversational AI | Chatbot | LLM Evaluation | AI Evaluation | AI Quality Analyst | Prompt Engineering | Generative AI | Remote

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Conversational AI Designer
Senior Conversational AI Designer

ZipStaff Inc. • Eden Prairie (MN)

Hybrid
USD 110,000 - 170,000
Chatbot Response Quality Analyst
Chatbot Response Quality Analyst

OpenTrain AI, Inc. • United States

Remote
USD 41,000 - 83,000
UX Conversation Designer, Vice President
UX Conversation Designer, Vice President

JP Morgan Chase • New York (NY)

On-site
USD 150,000 - 210,000
Conversational UX Writer
Conversational UX Writer

Think Company • Philadelphia

Remote
USD 90,000 - 120,000
Competitive compensation
Comprehensive benefits
Ongoing learning opportunities
AI Chatbot Quality Designer (Remote)
AI Chatbot Quality Designer (Remote)

Catapult Solutions Group • United States

Remote
USD 57,000 - 82,000
Sr. Conversational AI Designer
Sr. Conversational AI Designer

TEKsystems • United States

Remote
USD 110,000 - 131,000
Medical, dental & vision
401(k) Retirement Plan
Life Insurance
+4
Conversational AI & NLP Specialist
Conversational AI & NLP Specialist

Linda Werner & Associates • Myrtle Point (OR)

Remote
USD 120,000 - 180,000
Dental insurance
Health insurance
Health savings account
+4
Conversational Design Lead, Vice President
Conversational Design Lead, Vice President

J.P. Morgan Chase & Co. • New York (NY)

On-site
USD 180,000 - 280,000
Associate AI Conversation Designer (German)
Associate AI Conversation Designer (German)

JustAnswer • United States

Remote
USD 60,000 - 90,000
ChatBot QA
ChatBot QA

Matlen Silver • Charlotte (NC)

Hybrid
USD 61,992 - 68,880