Director, AI Evaluation & Quality for Product Experiences

Figma; New York (NY); On-site; USD 258,000 - 348,000; Full time

Benefits offered by this job

Equity

Health benefits

Parental leave

Wellness benefits

Paid time off

Job summary

Figma is seeking a Director, Research – AI Evals to own how we measure the quality of AI-powered experiences. You will define what is considered good, build evaluation frameworks, and turn results into signals used by product teams across the company.

You'll lead human and model-based evaluation, partner with Product, Design, Engineering, and Data Science, and shape evaluation as a strategic function in product decisions. This role can be remote within the United States.

Qualifications

• 10+ years in product, research, or closely related field; includes 2+ years in management.
• Hands-on experience owning the evaluation of AI/LLM-powered products.
• Expertise designing and running AI evaluation—rubrics, benchmarks, golden datasets.

Responsibilities

• Own AI evaluation methods and operations; define quality dimensions and measurements.
• Build evaluation frameworks, rubrics, golden datasets, and quality bars; combine human and automated approaches.
• Partner with engineering to stand up reproducible evaluation pipelines and regression testing.
• Produce clear readouts and dashboards to support go/no-go decisions.
• Socialize a shared definition of quality and advocate for evaluation as a strategic partner.
• Manage a small team with contractors and internal staff.

Skills

AI evaluation

Research leadership

Qual/quant methods

Stakeholder influence

Cross-functional collaboration

AI/LLM evaluation

Tools

Braintrust

LangSmith

DeepEval

Job description

Figma is seeking a Director, Research – AI Evals to own how we measure the quality of AI-powered experiences. You will define what is considered good, build evaluation frameworks, and turn results into signals used by product teams across the company.

You'll lead human and model-based evaluation, partner with Product, Design, Engineering, and Data Science, and shape evaluation as a strategic function in product decisions. This role can be remote within the United States.