AI Chatbot Quality Designer (Remote)

Catapult Solutions Group

United States

Remote

USD 57,000 - 82,000

Full time

27 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Catapult Solutions Group is seeking a Conversation Designer for an all-remote contract role (US, Pacific Time hours) focused on AI chatbot evaluations. You will build datasets, review responses, and score quality against rubrics, collaborating with product, engineering and data science teams.

The ideal candidate has 3–5 years in conversation design, strong analytical skills, and comfort using AI tools. This role offers a 12‑month contract with remote work from anywhere in the US.

Qualifications

  • 3–5 years of experience in conversation design or related AI evaluation
  • Experience evaluating chatbot responses for accuracy and relevance
  • Familiarity with structured content, rubrics and QA processes
  • Strong analytical and logical reasoning skills
  • Excellent written communication and attention to detail

Responsibilities

  • Create large datasets of realistic customer questions with verified correct answers
  • Develop question-and-answer pairs to evaluate chatbot performance
  • Review AI-generated responses for accuracy, relevance and quality
  • Compare chatbot answers against approved source content and documentation
  • Score responses using evaluation rubrics and design principles
  • Identify incorrect answers and recurring error patterns; report findings to Engineering
  • Calculate accuracy percentages and monitor performance trends
  • Provide actionable feedback to Engineering and product partners
  • Use AI tools to streamline evaluation workflows
  • Build simple tools or automations with AI-assisted tools when helpful

Skills

Conversation design
AI/LLM evaluation
Analytical thinking
Attention to detail
Pacific Time availability

Education

Bachelor's degree preferred

Tools

Claude Code
ChatGPT

Job description

Catapult Solutions Group is seeking a Conversation Designer for an all-remote contract role (US, Pacific Time hours) focused on AI chatbot evaluations. You will build datasets, review responses, and score quality against rubrics, collaborating with product, engineering and data science teams.

The ideal candidate has 3–5 years in conversation design, strong analytical skills, and comfort using AI tools. This role offers a 12‑month contract with remote work from anywhere in the US.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Conversational Designer
Conversational Designer

Catapult Solutions Group • United States

Remote
USD 57,000 - 82,000
Remote AI Chatbot Quality Analyst
Remote AI Chatbot Quality Analyst

Turing • United States

Remote
USD 34,000 - 55,000
Fully remote work
Flexible schedule
Global project exposure
AI Chatbot QA Analyst - Remote 16-Week Project
AI Chatbot QA Analyst - Remote 16-Week Project

Braintrust • United States

Remote
USD 26,000 - 37,000
Remote QA Analyst: AI Chatbot Quality Assessor
Remote QA Analyst: AI Chatbot Quality Assessor

Turing Enterprises, Inc. • San Francisco (CA)

Remote
USD 55,000 - 83,000
Fully remote
Flexible work environment
Global collaboration
Remote Chatbot Quality Analyst (Part-Time)
Remote Chatbot Quality Analyst (Part-Time)

OpenTrain AI, Inc. • United States

Remote
USD 41,000 - 83,000
Lead Conversation Designer for AI Quality & Golden Datasets
Lead Conversation Designer for AI Quality & Golden Datasets

Akraya Inc. • Mountain View (CA)

Hybrid
USD 62,000 - 69,000
Chatbot Response Quality Analyst
Chatbot Response Quality Analyst

OpenTrain AI, Inc. • United States

Remote
USD 41,000 - 83,000
Remote QA Analyst — AI Chatbot Evaluations (16-Week Contract)
Remote QA Analyst — AI Chatbot Evaluations (16-Week Contract)

Turing Enterprises, Inc. • United States

Remote
USD 41,000 - 62,000
Fully remote
Flexible schedule
Global collaboration
Remote AI Project Analyst — Chatbot Evaluation
Remote AI Project Analyst — Chatbot Evaluation

HumanitApp • United States

Remote
USD 55,000 - 83,000
Fully remote
Flexible work environment
Global collaboration
Remote Chatbot QA Specialist: Precision & Insights
Remote Chatbot QA Specialist: Precision & Insights

Turing • United States

Remote
USD 25,000 - 55,000
Fully remote
Flexible schedule
Global collaboration