Applied AI Researcher, Benchmarking

FDE Times

San Francisco (CA)

Hybrid

USD 150,000 - 250,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Equity
Health insurance
Flexible time off
401(k) plan
Wellness benefits
Lunch provided

Job summary

Distyl AI seeks researchers who redefine how AI is used in enterprise, beyond benchmarks. You’ll work across evaluation frameworks, measurement of reasoning depth, and real-world impact, building prototypes with agentic AI and ensemble approaches.

This role emphasizes practical AI usage, strong programming and data analysis, and delivering measurable research outcomes. Distyl AI values curiosity and pragmatism, offering a hybrid model with in-office collaboration and access to cutting-edge AI

Qualifications

  • Designed or maintained benchmarks, test suites, or experimental frameworks to measure model performance.
  • Statistical and analytical rigor to design fair, reproducible experiments and extract signal from noisy results.
  • Experience building with models and AI systems, not just training or fine tuning them.
  • Proven track record of research results through publications or demonstrable work.
  • Daily use of AI tools to accelerate workflows.

Responsibilities

  • Define evaluation frameworks capturing reasoning depth, interaction quality, reliability, and operational impact.
  • Investigate metrics shaping model behavior and establish methodologies for quantifying emergent capability.
  • Drive Distyl’s internal research priorities and industry standards.

Skills

Benchmark design
Statistical rigor
Model-based prototyping
Research results
Practical AI usage
Programming skills
Data analysis
Communication of impact

Tools

ChatGPT
Cursor
Perplexity

Job description

About Distyl AI

Distyl is an applied AI technology company partnering with the world’s most ambitious institutions to rearchitect critical operations for the frontier of AI. Our customers include the largest companies in telecom, healthcare, insurance, manufacturing, consumer goods, and global social organizations.

We research and deploy technologies that power AI-native operations — both for our partners and for Distyl itself. Our work spans research into self-constructing systems, the development of the most reliable execution of AI systems, and products that transform mission-critical workflows. As a result, Distyl's technologies affect some of the world's largest operations — from hundreds of millions of consumer interactions to tens of millions of supply chain transactions and millions of patient journeys.

Distyl is backed by leading investors including Lightspeed Venture Partners, Khosla Ventures, Coatue, DST Global, and the board-members of 20+ F500s.

What We Are Looking For

At Distyl we’re pushing the envelope of AI utilization in enterprise. This requires creative researchers who don’t just want to drive incremental improvements on benchmarks or optimize an existing process but instead are looking to creatively redefine how software is used.

Our researchers come from many academic backgrounds but have strong research track records, operate in an AI-native way, and would be bored staying on the rails of a traditional research org.

Key Responsibilities
  • The Benchmarking team defines how progress is measured. Researchers design evaluation frameworks that capture reasoning depth, interaction quality, reliability, and operational impact. They construct benchmarks that reflect real-world complexity. Their systems become the standard by which new architectures, techniques, and releases are judged.
  • Researchers in Benchmarking explore new paradigms for evaluating intelligent systems: adversarial robustness testing, longitudinal performance tracking, and human-in-the-loop assessment. They investigate how metrics shape model behavior and establish rigorous methodologies for quantifying emergent capability. Their insights drive both Distyl’s internal research priorities and industry-wide standards.
Who You Are
  • Experience Designing and Running Evaluations: You’ve built or maintained benchmarks, test suites, or experimental frameworks to measure model or system performance
  • Statistical and Analytical Rigor: You design fair, reproducible experiments and can extract signal from noisy empirical results
  • Experience Building with Models, Not Just Building Models: We develop intelligent systems using models rather than training or fine-tuning them. Ideal candidates have expertise in compound AI systems, agentic collaboration, and associated techniques (ensembling, ReAct, graph-of-thoughts, etc.)
  • Proven Track Record of Research Results: Whether you’ve published in top journals, posted amazing work on twitter, or somewhere else we want to see what you've done
  • Uses AI Every Day: Before you can revolutionize someone else’s workflow, you need to revolutionize yours. You should be using tools like ChatGPT, Cursor, and Perplexity to accelerate your workflow
  • Strong Programming and Data Analysis Skills: While you might not consider yourself a software engineer you need to be able to build prototypes of your ideas and then perform the experiments to prove the effectiveness to a F500 Head of AI
  • Biases Towards Showing vs Telling: Our customers want to see the power of AI today vs discuss the most elegant idea that will take 5 years to realize
What We Offer
  • The base salary range for this role is $150K – $250K, depending on experience, location, and level. In addition to base compensation, this role is eligible for meaningful equity, along with a comprehensive benefits package
  • 100% coverage of medical, dental, and vision insurance for employee and dependents
  • Flexible time off
  • Retirement and financial planning benefits, including access to pre-tax HSA, FSA, and commuter accounts, 401(k), and financial coaching resources
  • Comprehensive wellness benefits, including physical fitness, mental well-being, and fertility and family-building benefits through Carrot
  • Complimentary in-office lunches and snacks provided
  • Access to state-of-the-art AI models, generous usage of modern AI tools, and real-world business problems
  • Ownership of high-impact projects across top enterprises
  • A mission-driven, fast-moving culture that values curiosity, pragmatism, and excellence

Distyl has offices in San Francisco and New York. This role follows a hybrid collaboration model with 3+ days per week (Tuesday–Thursday) in‑office.

We believe diverse perspectives make our work stronger and more impactful. We are an equal opportunity employer and evaluate all applicants without regard to race, color, religion, sex, sexual orientation, gender identity or expression, national origin, age, disability, veteran status, or any other legally protected characteristic. We encourage candidates from all backgrounds to apply.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Applied AI Researcher, Benchmarking
Applied AI Researcher, Benchmarking

DISTYL • San Francisco (CA)

Hybrid
USD 150,000 - 250,000
Equity
Medical, dental, vision
Flexible time off
+4
Applied AI Researcher, System Self-Improvement
Applied AI Researcher, System Self-Improvement

FDE Times • San Francisco (CA)

Hybrid
USD 150,000 - 250,000
Base salary 150k-250k USD per year
Meaningful equity
Medical,dental,vision insurance
+4
Applied AI Researcher, System Discovery
Applied AI Researcher, System Discovery

FDE Times • San Francisco (CA)

Hybrid
USD 150,000 - 250,000
Equity
Comprehensive benefits
Flexible time off
+4
Applied AI Researcher, System Discovery
Applied AI Researcher, System Discovery

DISTYL • San Francisco (CA)

Hybrid
USD 150,000 - 250,000
Equity
Medical/dental/vision insurance
Flexible time off
+3
Research Engineers, Post-Training
Research Engineers, Post-Training

DISTYL • San Francisco (CA)

Hybrid
USD 150,000 - 250,000
Equity
Insurance coverage
Flexible time off
+5
Research Engineers, Post-Training
Research Engineers, Post-Training

DISTYL • New York (NY)

Hybrid
USD 150,000 - 250,000
Equity compensation
Medical insurance
Flexible time off
+7
Research Engineers, Data
Research Engineers, Data

DISTYL • New York (NY)

Hybrid
USD 150,000 - 250,000
Equity
Medical, dental, vision insurance
Flexible time off
+1
Applied AI Researcher, Multi-Agent Systems
Applied AI Researcher, Multi-Agent Systems

FDE Times • San Francisco (CA)

Hybrid
USD 150,000 - 250,000
Equity
Comprehensive benefits
Flexible time off
+1
Research Engineers, Agents
Research Engineers, Agents

DISTYL • New York (NY)

Hybrid
USD 150,000 - 250,000
Equity
Medical/Dental/Vision
Flexible time off
+2
Research Engineers, Agents
Research Engineers, Agents

DISTYL • San Francisco (CA)

Hybrid
USD 150,000 - 250,000
Equity
Medical, dental & vision insurance
Flexible time off
+3