Researcher, AI Evaluations

Bazeta; United States; Remote; USD 140,000 - 230,000; Full time

Job summary

Figma is hiring a Senior Researcher, AI Evaluations to join AI Insights within Figma’s Research team. This senior IC role reports to the AI Insights lead and focuses on evaluating AI products, building robust rubrics, and advancing evaluation methods.

You will own evaluation research for priority AI workstreams, translate findings into product improvements, and collaborate with researchers and engineers to scale rigorous assessments. This role can be remote in the United States.

Qualifications

• 7+ years of experience in research, applied research, or related field and proven ability to lead evaluation projects.
• Experience evaluating AI products and translating findings into model/product improvements.
• Expertise in designing human evaluations, scoring rubrics, datasets and benchmarks.
• Strong qualitative and quantitative research skills and ability to interpret uncertainty.
• Excellent problem-solving and stakeholder management skills.

Responsibilities

• Own evaluation research for priority AI workstreams.
• Define quality criteria, design studies, analyze results and recommend actions.
• Build rubrics, datasets, benchmarks reflecting user needs and expert judgment.
• Collaborate with AI Research and Modeling Platform engineers to integrate methods.
• Explain quality, failure patterns, and uncertainty; guide model and product decisions.
• Validate automated graders against human ratings and identify bias or gaps.
• Share reusable methods and mentor peers.

Skills

AI evaluation

Research leadership

Rubrics design

Statistical analysis

Qualitative analysis

Cross-functional

Tools

Braintrust

LangSmith

DeepEval

Job description

# Researcher, AI EvaluationsUnited States3 hours agoID 5627586Price on request## DetailsEmployment type: Full-timeRemote: YesCompany: Figma## DescriptionFigma is growing our team of passionate creatives and builders on a mission to make design accessible to all. Figma’s platform helps teams bring ideas to life—whether you’re brainstorming, creating a prototype, translating designs into code, or iterating with AI. From idea to product, Figma empowers teams to streamline workflows, move faster, and work together in real time from anywhere in the world. If you’re excited to shape the future of design and collaboration, join us! We’re hiring a Researcher, AI Evaluations to join AI Insights within Figma’s Research team. This is a senior individual contributor role with no direct reports, reporting to the AI Insights lead and working directly with AI Research within AI Platform. You’ll strengthen the team’s model, agent system, and evaluation work through hands-on studies, clear rubrics, and reliable measures of quality that reflect user needs, and your findings will guide model and product improvements. This is a full time role that can be held from one of our US hubs or remotely in the United States. What you’ll do at Figma: • Own evaluation research for priority AI Research workstreams. Define quality criteria, design studies, analyze results, and recommend actions • Build rubrics, representative evaluation datasets, benchmarks, and quality thresholds that reflect user needs and expert design judgment • Work with AI Research and Modeling Platform engineers to integrate validated methods into repeatable evaluations and regression tests • Explain quality, failure patterns, and uncertainty. Recommend actions that guide model selection, model improvements, product priorities, and release decisions • Validate automated graders against human ratings, including LLM-as-judge methods that use a large language model to score outputs. Identify bias, gaps, and uncertainty • Connect user research with AI Research’s model development work. Share reusable methods, review work, and mentor peers We’d love to hear from you if you have: • 7+ years of experience in research, applied research, or a related field, with a proven track record of independently leading complex evaluation projects and delivering measurable impact • Hands-on experience evaluating AI products, with demonstrated ability to translate evaluation findings into meaningful model and product improvements • Expertise in designing human evaluations, scoring rubrics, datasets, and benchmarks, including measuring inter-rater agreement and validating automated graders • Strong qualitative and quantitative research skills, with the ability to analyze data, interpret uncertainty, and draw actionable insights into model behavior and performance • Excellent problem-solving and stakeholder management skills, with the ability to navigate ambiguity, prioritize risks, balance speed with rigor, and deliver recommendations that influence cross-functional decisions While it’s not required, it’s an added plus if you also have: • Experience integrating automated evaluations and regression testing into engineering workflows, with familiarity in tools such as Braintrust, LangSmith, DeepEval, or equivalent platforms • Proven ability to develop scalable, reusable evaluation frameworks and drive the adoption of consistent quality standards across teams • Background in product design, product management, data science, or engineering, with familiarity in Figma or similar design and developer tools • Strong statistical expertise in sampling, significance testing, and agreement metrics, with experience conducting experiments or analyzing performance metrics for AI-powered features • Advanced training in a quantitative or behavioral discipline, gained through formal education or equivalent hands-on experience At Figma, one of our values is Grow as you go. We believe in hiring smart, curious people who are excited to learn and develop their skills. If you’re excited about this role but your past experience doesn’t align perfectly with the points outlined in the job description, we encourage you to apply anyways. You may be just the right candidate for this or other roles.