AI OPERATIONS SPECIALIST (contract)

Airbnb; San Francisco (CA); On-site; USD 85,000 - 105,000; Full time

Benefits offered by this job

Medical

Dental

Vision

401K

Job summary

Airbnb is seeking a hands-on AI operations leader to drive scalable AI workflows across teams. You will design prompts, build pipelines, and maintain quality as models evolve, enabling product and business teams to deploy AI solutions with measurable impact.

You will work with product, Legal and Policy to translate requirements into repeatable processes, and you’ll document learnings to accelerate future AI initiatives. A true self-starter thrives here.

Qualifications

• 5-7 years in operations, program management, analytics, consulting or applied AI.
• Experience building AI workflows or automations with measurable impact.
• Proficient with SQL; notebooks help.
• Curious about how work gets done and measurable processes.
• Experience with quality, QA or audits.
• Strong written communication with business leads, PMs and engineers.
• Self-directed with minimal guidance; no strict technical background required.
• Training available for evaluating LLMs.

Responsibilities

• Build the AI workflow: write prompts, set up the pipeline, and keep it at the agreed quality level.
• Write prompts and manage model-in-the-loop testing.
• Build test sets, measure accuracy, and calibrate LLM judges.
• Support product teams in go/no-go decisions for AI features.
• Create labeling guidelines and produce human-verified labels on data.
• Run labeling at scale with vendor agents and verify adherence to metrics.
• Document learnings to enable other teams to productionize quickly.
• Keep methods current as models and tools evolve; retire ineffective approaches.
• Own assigned workstreams from setup through quality checks; coordinate with teams.

Skills

Operations experience

Applied AI

Analytical mindset

Strong communication

SQL

Python

Tools

API experience

Job description

Airbnb was born in 2007 when two Hosts welcomed three guests to their San Francisco home and has since grown to over 4 million Hosts who have welcomed more than 1 billion guest arrivals in more than 100, 000 cities and towns located in 220+ countries and regions across the globe. Every day, Hosts offer unique stays and one-of-a-kind activities that make it possible for guests to experience the world in a more authentic, connected way.

Airbnb was born with a creative spirit, and it’s one of the most defining parts of our culture. We use our curiosity and imagination to come up with unconventional solutions. These are the ideas at the core of Airbnb:

• Our community is based on connection and belonging.
• Our creativity allows us to imagine new possibilities for people.
• Our responsibility is to all of our stakeholders.
• It takes a unified team committed to our core values to embody these ideas, and our fast-moving team is committed to leading as a 21st century company.

The Community you will join:

We're the AI & ML Enablement team in Supply Tech. We help teams across Airbnb use AI to do real work, and prove that it works. Right now, we're helping business and field teams use AI to take on the review and quality work they run, and helping product teams understand whether an AI feature works before it reaches our users. Day to day, we move manual review processes to AI and put people where their judgment matters most. We design new workflows so AI does the work from the start. We also build the labeled datasets that show whether the AI is right, and we keep everything running as models and policies change. Our methods and projects change as the models do, and we expect the scope of this role to grow with them.

The difference you will make:

You’ll work alongside our team on whichever problems AI could take on, whether that’s a business team with a process that's slow, expensive or inconsistent, or a product team getting an AI feature ready to launch. The job looks much the same either way. You start with how the work really gets done, help build an AI version of it, and test it until you trust the results. Then you hand it off and use what worked on the next problem. The mix of projects will keep changing as AI takes on more of the work, so we care more about how you work than about a fixed list of projects. This is a hands‑on role. You help build the workflow, ship it, and stay on to make sure it works.

A typical day:

• Help the team decide which projects will drive the largest business impact, and make the case for the ones worth doing.
• Sit with the team that owns a problem, map how the work gets done today, and work out how AI can do it and where people step in for the hard cases.
• Build the AI workflow: write the prompts, set up the pipeline, and keep it at the agreed quality level, watching for drift once it's live.
• Build the test sets, measure accuracy, and calibrate LLM judges against human reviewers.
• Support product teams in agreeing on what "good enough to launch" means for an AI feature, and bring the evidence for the go/no-go call.
• Turn unclear policy into criteria a reviewer can apply consistently, working with product, Legal and Policy.
• Work with subject matter experts to write labeling guidelines and produce human-verified labels on real data.
• Run labeling at scale with vendor agents and check their work against agreement metrics and your own spot checks.
• Make what you build reusable and document what you learn, so the next team gets to production faster.
• Keep our methods current as models, tools and vendor agents change, and retire what stops working.
• Own the workstreams you're assigned, from setup through quality checks. Model infrastructure and product roadmaps stay with the teams that own them.

Your expertise:

• 5-7 years in operations, program management, analytics, consulting or applied AI, where you were responsible for results.
• You use AI every day and have built something with it that has actually moved the needle, such as a workflow, an automation or an analysis. You know where it breaks, and you keep pushing it further.
• You're comfortable with numbers: accuracy, precision and recall, sampling. SQL and notebooks help. If you haven't used them yet, you'll pick them up quickly with AI assistance.
• You're curious about how work actually gets done, including the awkward exceptions, and you've taken a messy process and made it measurable.
• You've worked on quality in some form, such as review guidelines, QA or audits.
• You write clearly and can work with business leads, PMs and engineers in the same week.
• You're fine working with limited direction, and you try things yourself instead of waiting for someone to build them. A technical background isn't required.
• You don't need to have designed evaluations or calibrated LLM judges before. We'll teach you, and you'll work with a team that has done it many times.

Preferred Experience

• Hands‑on SQL, Python or API experience.
• Trust and safety, content moderation, annotation operations or other human‑in‑the‑loop work at scale.
• Fine‑tuning models or calibrating LLM judges.
• Background in a two‑sided marketplace, travel, or another operations‑heavy business. Experience setting up intake and prioritization for a new team.

Belonging & Accommodations

Airbnb is committed to working with the best and brightest people from the broadest talent pool possible. We believe a diversity of ideas fosters innovation and engagement, and allows us to attract the best people, and to develop the best products, services and solutions. All qualified individuals are encouraged to apply. If you need assistance, or reasonable accommodation during the application and recruiting process, please connect with your talent sourcing partner.

Pay Rate Range: $62-$76/hr

• Medical
• Dental
• Vision
• 401K