Creative AI Research Project Lead

Contra; San Francisco (CA); On-site; USD 150,000 - 200,000; Full time

Benefits offered by this job

Company laptop

401k matching

Job summary

Contra Labs is a human-centered research lab exploring how creatives use AI and how tools can better serve creative work. We partner with top labs and product companies to build benchmarks, evaluation layers, and post-training data solutions that reflect real creative workflows.

You will own post-training dataset creation, benchmarking programs, and ongoing evaluation partnerships across diverse teams—bridging clients, creative experts, and engineers to ship rigorous, practical results.

Qualifications

• 3+ years designing studies, running evaluations, analyzing data, and presenting findings.
• Experience with data labeling, annotation, human-in-the-loop systems, or AI and ML evaluation workflows.
• A track record managing several concurrent projects, including annotation, evaluation, or QA work.
• A real point of view on creative quality and what 'good' looks like.

Responsibilities

• The methodology. Annotation protocols, rubrics, and evaluation designs for creative and multimodal outputs.
• Delivery. Scope, evaluator staffing, quality routing, timelines, and the client relationship, from kickoff to final readout.
• Quality. Data quality, inter-rater reliability, and real-time course correction when something feels off. You hold the bar, and you hold the levers to fix it.
• Analysis and storytelling. Qualitative and quantitative analysis turned into clear findings, recommendations, reports, and workshops that clients act on.
• The systems behind it. You turn repeated work into clean, measurable process, and you partner with engineering on the tooling that makes that process faster and more reliable.
• The narrative. Case studies and benchmark publications that sharpen how Contra Labs is understood in the market.

Skills

Study design

Evaluation design

Data analysis

Findings presentation

Data labeling

AI/ML evaluation workflows

Project management

Creative quality judgment

Client communication

Education

Master's or PhD in HCI / related field

Design research degree / equivalent industry depth

Job description

Contra Labs is a human-centered research lab exploring how creatives use AI and how tools can better serve creative work. We partner with top labs and product companies to build benchmarks, evaluation layers, and post-training data solutions that reflect real creative workflows.

You will own post-training dataset creation, benchmarking programs, and ongoing evaluation partnerships across diverse teams—bridging clients, creative experts, and engineers to ship rigorous, practical results.