AI Trainer

Pay Ready, Inc.; United States; Remote; USD 110,000 - 160,000; Full time

Benefits offered by this job

100% Remote

Flexible hours

Job summary

Pay Ready, Inc. is seeking an AI Operations QA Gatekeeper to review agent conversations, diagnose discrepancies, and validate fixes before deployment. You will test scenarios, document trace IDs, and determine whether wording, tools, or rules caused issues.

This role requires strong attention to regulatory constraints and precise written communication. You will work with engineers, Compliance, and Operations to maintain a compliant release process, update decision matrices, and ensure accurate

Qualifications

• Highly detail-oriented and able to apply regulatory rules to human language in conversations.
• Experience with QA, testing, and release gating in AI-assisted workflows.
• Ability to clearly document findings, traces, IDs, and recommended actions for technical audiences.

Responsibilities

• Review agent conversations and determine why outcomes differ from expectations.
• Test changes in scenarios and document traces and results.
• Escalate blockers and separate findings that ship with a ticket.
• Write prompt or skill changes and run evaluations before production.

Skills

Regulatory compliance

Attention to detail

Written communication

Quality assurance

Release management

Tools

LangSmith

Looker

Linear

Notion

Slack

Job description

Pay Ready handles payment and recovery for property management companies, covering rent for current residents and balances owed after move-out. Most of that contact now runs through AI agents over text, email and recorded voice drops: around 40,000 agent runs on an average day, more than 70,000 on the heaviest. Debt collection is regulated, so every one of those messages has to be accurate and has to stay inside FDCPA, TCPA and state rules. The AI Operations team is the human layer that checks that and fixes what is wrong.

THE JOB

You read what the agents said to residents yesterday, decide what was wrong, fix what you can fix in the agent's own instructions, and prove the fix before it ships. You are the release gate. When a change is ready for production, you test it scenario by scenario, and your results decide whether it goes.

Most of the day is judgment. One failure has topped the daily report every day for weeks. The agent tells a resident "I've submitted this for review." The trace shows no tool call, so nothing was submitted. You work out whether that is a wording problem, a broken tool, or a rule nobody wrote down, and you say which. You are the one reading the conversations, so you see what the traces alone do not.

DAY-TO-DAY

• The daily production read. Pull the previous day's numbers from the Looker dashboard and the review queue, then post one report: volume, outbound and inbound success rates, escalations by type, held messages by reason, the five failure modes that mattered with counts, and four things to fix that day with a target number on each. The summary goes to the team channel, the full write-up to Notion.
• The review queue. Messages that fail the compliance auditor are held and posted as cards for a decision. Several hundred land on a normal day, sometimes more than a thousand. Work them against a written decision matrix you maintain, and pull out the ones that are not queue hygiene: a self-harm flag, an attorney or bankruptcy disclosure, a cease-and-desist, a resident who paid and whose payment never got recorded.
• Release validation. Run the scenarios a change touches against a test account, posing as a resident, and write up each one with the trace ID, what the agent said, and the result. Name the blockers that hold the release and keep them separate from the findings that ship with a ticket. Call out a pass that only held because a downstream guardrail caught something, because that pass will not survive the guardrail changing. Re-run to tell a real defect from a one-off, and discount a finding when your own test setup caused it.
• Prompt and skill changes. Most of the agent's instructions live in a content hub you edit directly: write the change, run the dataset evaluation, get Compliance sign-off when the wording carries legal risk, and promote it from staging to production yourself. The rest of the agent's behavior lives in the codebase and ships on an engineer's deploy. Knowing which is which is part of the job.
• Evals and baselines. Build the golden and evaluation datasets each skill is scored against, keep the automated graders agreeing with what a human reviewer would say, and run them against production traces.
• Undocumented rules. When a conversation exposes a rule nobody wrote down, write it into the dated compliance ledger. Rules that carry regulatory weight go to Compliance for sign-off; product rules go to the product lead. Neither goes live without that.

WHAT YOU DECIDE

How a held message is dispositioned and which rule it cites, which action code fits, what goes into an evaluation dataset, how a prompt or skill is worded, what the day's priorities are, and whether a release clears your validation.

WHAT YOU ESCALATE

Any change to consent, suppression, or how an opt-out is honored. Any new resident-facing wording on a legal-notice path. Anything that needs a backend change or a tool that does not exist yet. Bring each one with a recommendation and the evidence behind it.

WHO YOU WORK WITH

You report into product. Day to day you work with the engineers on the agent, with Compliance on anything touching disclosures and dispute rights, and with the operations team that handles files the agent escalates. You own one row on the weekly product scorecard.

WHAT WE'RE LOOKING FOR

• You read closely and do not round off. The gap between "the agent said it recorded the payment" and "the payment was recorded" is the whole job.
• You can hold a regulated rule in your head and apply it to messy human language. Prior work in collections, property management, compliance, QA or any regulated operation counts for more here than AI experience.
• You write clearly for a technical audience: the finding, the trace ID, what you think caused it, and what you want done.
• You are comfortable being the person who blocks a release, and comfortable retracting that when you turn out to be wrong.
• You can work a repetitive daily queue without going on autopilot. The one you skim is the one that mattered.
• You are willing to learn the tooling. It is LangSmith for traces, datasets and evaluators, Looker for reporting, Linear for tickets, Notion and Slack for everything else. You do not need to have used any of them.

Helpful, not required

Property management or collections background. SQL or BI reporting. Experience writing SOPs that other people actually follow. Hands-on time with prompts, evaluation sets, or conversation QA.

First 90 days

By day 30, posting the daily report and working the review queue without help. By day 60, writing prompt and skill changes and running the evaluation before they ship. By day 90, your sign-off is the validation gate on a release.

WHY JOIN?

These agents are the ones having the conversation when someone is behind on what they owe. Whether it goes well comes down to the rules you write and the mistakes you catch. A change you make this afternoon can be in front of residents tomorrow, and you can count the difference in the next morning's report. The team is small and there is no layer between you and the decisions.

Equal opportunity

Pay Ready is an equal opportunity employer. We consider every applicant without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, veteran status, or any other protected characteristic. If you need an accommodation at any point in the hiring process, tell us and we will arrange it.

Work Location:

100% Remote. This position is fully remote and can be performed from any location within the United States. You are not required to live near or commute to an office.