AI Operations Specialist

Horizontal Talent; San Francisco (CA); Remote; USD 54,000 - 103,000; Full time

Benefits offered by this job

Medical benefits

Dental benefits

Vision benefits

Retirement plan

Job summary

Horizontal Talent is seeking an AI Operations Specialist for a fully remote role. You will drive high-impact AI projects by mapping workflows, building prompts and pipelines, and ensuring quality as models and vendors evolve.

You will collaborate with product, Legal and Policy, producing clear labeling guidelines and data for scalable labeling at scale. Strong communication and autonomous execution are essential for success in this role.

Qualifications

• 5-7 years in operations, program management, analytics, consulting or applied AI, with results.
• You use AI daily and have built something with it that moved the needle, e.g., a workflow, automation or analysis.
• You're comfortable with numbers: accuracy, precision and recall; SQL and notebooks help.
• You're curious about how work gets done, including exceptions, and can measure processes.
• You've worked on quality in some form: QA, audits, or guidelines.
• Clear written communication with business leads, PMs and engineers.
• You can work with limited direction; a technical background isn't required.
• We will teach you calibration of LLM judges and you’ll work with a team that has done it many times.

Responsibilities

• Decide which projects will drive the largest business impact and build the case for them.
• Map current processes with the team and design AI-enabled workflows.
• Create AI prompts, pipelines, and monitor quality and drift post-launch.
• Build test sets, measure accuracy, and calibrate LLM judges.
• Help product teams agree on launch criteria for AI features with evidence.
• Translate policy into concrete reviewer criteria with cross-functional partners.
• Collaborate with SMEs to write labeling guidelines and produce human-labeled data.
• Oversee labeling at scale with vendors and verify against metrics.

Skills

Operations experience

AI workflow design

Analytics & metrics

SQL & notebooks

Stakeholder communication

Autonomy

Tools

Python

APIs

SQL

Job description

AI Operations Specialist Remote

12 months

A typical day

• Help the team decide which projects will drive the largest business impact, and make the case for the ones worth doing.
• Sit with the team that owns a problem, map how the work gets done today, and work out how AI can do it and where people step in for the hard cases.
• Build the AI workflow: write the prompts, set up the pipeline, and keep it at the agreed quality level, watching for drift once it's live.
• Build the test sets, measure accuracy, and calibrate LLM judges against human reviewers.
• Support product teams in agreeing on what "good enough to launch" means for an AI feature, and bring the evidence for the go/no-go call.
• Turn unclear policy into criteria a reviewer can apply consistently, working with product, Legal and Policy.
• Work with subject matter experts to write labeling guidelines and produce human-verified labels on real data.
• Run labeling at scale with vendor agents, and check their work against agreement metrics and your own spot checks.
• Make what you build reusable and document what you learn, so the next team gets to production faster.
• Keep our methods current as models, tools and vendor agents change, and retire what stops working.
• Own the workstreams you're assigned, from setup through quality checks. Model infrastructure and product roadmaps stay with the teams that own them.

Skills

• 5-7 years in operations, program management, analytics, consulting or applied AI, where you were responsible for results.
• You use AI every day and have built something with it that has actually moved the needle, such as a workflow, an automation or an analysis. You know where it breaks, and you keep pushing it further.
• You're comfortable with numbers: accuracy, precision and recall, sampling. SQL and notebooks help. If you haven't used them yet, you'll pick them up quickly with AI assistance.
• You're curious about how work actually gets done, including the awkward exceptions, and you've taken a messy process and made it measurable.
• You've worked on quality in some form, such as review guidelines, QA or audits.
• You write clearly and can work with business leads, PMs and engineers in the same week.
• You're fine working with limited direction, and you try things yourself instead of waiting for someone to build them. A technical background isn't required.
• You don't need to have designed evaluations or calibrated LLM judges before. We'll teach you, and you'll work with a team that has done it many times.

Preferred Experience

• Hands-on SQL, Python or API experience.
• Trust and safety, content moderation, annotation operations or other human-in-the-loop work at scale.
• Fine-tuning models or calibrating LLM judges.
• Background in a two-sided marketplace, travel, or another operations-heavy business.
• Experience setting up intake and prioritization for a new team.

We offer competitive compensation and benefits including medical, dental, vision, and retirement.

The pay range for this role is $39 - $75 per hour based on qualifications and experience.