Remote AI Release Gatekeeper & Compliance QA

Pay Ready, Inc.; United States; Remote; USD 110,000 - 160,000; Full time

Benefits offered by this job

100% Remote

Flexible hours

Job summary

Pay Ready, Inc. is seeking an AI Operations QA Gatekeeper to review agent conversations, diagnose discrepancies, and validate fixes before deployment. You will test scenarios, document trace IDs, and determine whether wording, tools, or rules caused issues.

This role requires strong attention to regulatory constraints and precise written communication. You will work with engineers, Compliance, and Operations to maintain a compliant release process, update decision matrices, and ensure accurate

Qualifications

• Highly detail-oriented and able to apply regulatory rules to human language in conversations.
• Experience with QA, testing, and release gating in AI-assisted workflows.
• Ability to clearly document findings, traces, IDs, and recommended actions for technical audiences.

Responsibilities

• Review agent conversations and determine why outcomes differ from expectations.
• Test changes in scenarios and document traces and results.
• Escalate blockers and separate findings that ship with a ticket.
• Write prompt or skill changes and run evaluations before production.

Skills

Regulatory compliance

Attention to detail

Written communication

Quality assurance

Release management

Tools

LangSmith

Looker

Linear

Notion

Slack

Job description

Pay Ready, Inc. is seeking an AI Operations QA Gatekeeper to review agent conversations, diagnose discrepancies, and validate fixes before deployment. You will test scenarios, document trace IDs, and determine whether wording, tools, or rules caused issues.

This role requires strong attention to regulatory constraints and precise written communication. You will work with engineers, Compliance, and Operations to maintain a compliant release process, update decision matrices, and ensure accurate