Ads Harness Engineer

Circle Labs; San Francisco (CA); On-site; USD 120,000 - 170,000; Full time

Benefits offered by this job

Competitive pay and equity

Health, dental, and vision coverage

Access to frontier AI Engines

Job summary

Circle Labs, based in San Francisco, seeks a researcher/engineer to build experiments and evaluation tools assessing advertising usefulness in shared conversations on shapes.inc.

You will study intent, ad matching and generated presentation, turning questions into repeatable evaluations to distinguish quality improvements from single-metric changes.

Qualifications

• Strong programming skills and experience building reliable tools for experiments.
• Ability to turn ambiguous quality questions into measurable tests.
• Experience evaluating AI systems, search or recommendations.
• Clear communication of findings to product and engineering teams.
• Understanding of experimental design and bias in evaluation data.

Responsibilities

• Build a reusable evaluation harness for intent interpretation and ad matching.
• Create evaluation cases covering group dynamics and ad matching scenarios.
• Define measures of relevance, factual support, repetition, and fit.
• Run controlled comparisons of prompts and matching approaches.
• Connect offline findings with production feedback and costs.
• Collaborate with Ads Lead on regression checks and rollout recommendations.

Skills

Programming

Experimentation

Data analysis

AI evaluation

Communication

Tools

Python

SQL

Experimentation tools

Job description

About Circle Labs

Circle Labs is the company behind shapes.inc, a place to collaborate. We’re building multiplayer AI: shapes are collaborative assistants that take part in group conversations and do things with you and your people. Our next work includes helping groups set shared goals, delegate to specialist shapes and external agents, and follow progress through to completion. Making things, figuring things out, and sharing plans give people more reasons to spend time together.

We’re a small team in San Francisco. We use the product every day, talk to users, and work across design, engineering, and research.

About the role

Build the experiments and tools that tell us whether advertising is useful in a shared conversation on shapes.inc. Working with our team in San Francisco, you will study how intent, candidate ads and generated presentation fit the moment when people are spending time or doing things together. You will turn those questions into repeatable evaluations and help research and engineering distinguish meaningful quality improvements from changes in a single metric.

What you’ll do

• Build a reusable evaluation harness for intent interpretation, ad matching and generated presentation, with versioned inputs and reproducible results.
• Create evaluation cases from authorized sources that cover different preferences within a group, changing plans, shared purchase decisions, repeated recommendations, and situations without a useful ad match.
• Define and calibrate measures of relevance, factual support, repetition and conversational fit using both human review and automated checks.
• Run controlled comparisons of prompts, AI Engines and matching approaches; inspect disagreements and investigate why a result changed.
• Connect offline findings with production feedback and serving outcomes while accounting for missing inventory, provider failures, latency and evaluation cost.
• Work with the Ads Lead to turn research findings into regression checks and practical rollout recommendations.

What you’ll bring

• Strong programming skills and experience building reliable tools for experiments, data analysis or system evaluation.
• Ability to turn an ambiguous quality question into a measurable test and explain what the result does and does not establish.
• Practical understanding of experimental design, sampling and sources of bias in evaluation data.
• Experience evaluating AI systems, search, recommendations or another product where quality requires more than a simple correct answer.
• Careful judgment when working with conversation data and the ability to communicate findings clearly to product and engineering teams.

Nice to have

• Experience with ad relevance, contextual matching or recommendation evaluation.
• Experience building human review workflows and calibrating automated evaluators against human judgments.
• Familiarity with prompt experiments, generated-content evaluation or measuring quality in conversations with multiple people.

Benefits

• Competitive pay and equity
• Health, dental, and vision coverage
• Access to frontier AI Engines