AI Operations Specialist

Horizontal Talent; San Francisco (CA); Remote; USD 90,000 - 130,000; Full time

Job summary

Horizontal Talent seeks an AI Operations Specialist for a remote, 12-month engagement. You will help identify AI projects with the largest business impact, map current processes, and design AI-enhanced workflows with human-in-the-loop involvement where needed.

You will build prompts, pipelines, and labeling guidelines, run large-scale labeling with vendors, and calibrate AI judgments against human reviews. Strong ops background and AI experience are required, with familiarity in SQL or Python

Qualifications

• 5-7 years in operations, program management, analytics, consulting or applied AI with responsibility for results.
• Experience using AI to move business metrics via workflows, automation or analysis.
• Comfort with numbers: accuracy, precision, recall, sampling; SQL and notebooks helpful.
• Curiosity about how work gets done, and ability to measure processes and improve them.
• Experience with quality controls such as QA or audits.
• Clear written communication and ability to work with business leads, PMs and engineers.

Responsibilities

• Assist in deciding which AI projects will drive business impact and justify them.
• Map current processes and design AI-enabled workflows with human-in-the-loop where needed.
• Write prompts, set up pipelines, and monitor quality to prevent drift.
• Create test data and calibrate model judgments against human reviews.
• Define go/no-go criteria with product and policy teams and provide evidence for decisions.
• Write labeling guidelines with SMEs and produce human-verified labels at scale.

Skills

Operations

Program management

Analytics

Applied AI

Communication

Independent work

Quality assurance

Tools

SQL

Python

APIs

Job description

AI Operations Specialist Remote 12 Months

A typical day:

• Help the team decide which projects will drive the largest business impact, and make the case for the ones worth doing.
• Sit with the team that owns a problem, map how the work gets done today, and work out how AI can do it and where people step in for the hard cases.
• Build the AI workflow: write the prompts, set up the pipeline, and keep it at the agreed quality level, watching for drift once it’s live.
• Build the test sets, measure accuracy, and calibrate LLM judges against human reviewers.
• Support product teams in agreeing on what “good enough to launch” means for an AI feature, and bring the evidence for the go/no-go call.
• Turn unclear policy into criteria a reviewer can apply consistently, working with product, Legal and Policy.
• Work with subject matter experts to write labeling guidelines and produce human-verified labels on real data.
• Run labeling at scale with vendor agents, and check their work against agreement metrics and your own spot checks.
• Make what you build reusable and document what you learn, so the next team gets to production faster.
• Keep our methods current as models, tools and vendor agents change, and retire what stops working.
• Own the workstreams you’re assigned, from setup through quality checks. Model infrastructure and product roadmaps stay with the teams that own them.

Skills

• 5-7 years in operations, program management, analytics, consulting or applied AI, where you were responsible for results.
• You use AI every day and have built something with it that has actually moved the needle, such as a workflow, an automation or an analysis. You know where it breaks, and you keep pushing it further.
• You’re comfortable with numbers: accuracy, precision and recall, sampling. SQL and notebooks help. If you haven’t used them yet, you’ll pick them up quickly with AI assistance.
• You’re curious about how work actually gets done, including the awkward exceptions, and you’ve taken a messy process and made it measurable.
• You’ve worked on quality in some form, such as review guidelines, QA or audits.
• You write clearly and can work with business leads, PMs and engineers in the same week.
• You’re fine working with limited direction, and you try things yourself instead of waiting for someone to build them. A technical background isn’t required.
• You don’t need to have designed evaluations or calibrated LLM judges before. We’ll teach you, and you’ll work with a team that has done it many times.

Preferred experience

• Hands-on SQL, Python or API experience.
• Trust and safety, content moderation, annotation operations or other human-in-the-loop work at scale.
• Fine-tuning models or calibrating LLM judges.
• Background in a two-sided marketplace, travel, or another operations-heavy business.
• Experience setting up intake and prioritization for a new team.

For those that join the team, we offer competitive compensation and benefits including medical, dental, vision, and retirement.