Enterprise AI Benchmarking Architect & Evaluation Lead

Distyl; San Francisco (CA); Hybrid; USD 150,000 - 250,000; Full time

Benefits offered by this job

Equity

Medical, dental, vision

Flexible time off

401(k) and financial coaching

Wellness benefits

Complimentary in-office lunches

Access to AI models and tools

Job summary

Distyl AI is a frontier AI company seeking researchers to redefine how software is used in enterprise environments. You will design evaluation benchmarks and explore novel paradigms for intelligent systems, shaping metrics, reliability, and operational impact.

The role emphasizes hands-on prototyping, rigorous analysis, and publication-ready results. Distyl offers equity, a hybrid work model, and access to leading AI models for enterprise-scale problems.

Qualifications

• Experience designing and running evaluations.
• Strong statistical and analytical rigor.
• Experience building with models, not just building models.
• Proven track record of research results.
• Uses AI daily with tools like ChatGPT, Cursor, Perplexity.
• Strong programming and data analysis skills.
• Bias toward showing results, not just ideas.

Responsibilities

• Define evaluation frameworks and benchmarks.
• Explore evaluation paradigms for intelligent systems.
• Measure model behavior and establish methodologies for emergent capabilities.

Skills

Benchmarking design

Statistical rigor

AI systems

Research track record

Practical AI tools

Prototyping

Tools

ChatGPT

Cursor

Perplexity

Python

Job description

Distyl AI is a frontier AI company seeking researchers to redefine how software is used in enterprise environments. You will design evaluation benchmarks and explore novel paradigms for intelligent systems, shaping metrics, reliability, and operational impact.

The role emphasizes hands-on prototyping, rigorous analysis, and publication-ready results. Distyl offers equity, a hybrid work model, and access to leading AI models for enterprise-scale problems.