Founding AI Learning & Evaluation Scientist — Equity

Socket.dev; Beverly Hills (CA); On-site; USD 180,000 - 280,000; Full time

Benefits offered by this job

Daily team dinner provided in-office

Job summary

Socket.dev in Beverly Hills is seeking a principal AI evaluation scientist to own the measurement layer under model training and learning science across StudyFetch and Honen. You will design evals, run benchmarks, and ensure safety and educational quality.

This founding-team role offers direct collaboration with decision-makers, end-to-end analytics ownership, and a culture of rigorous causality, transparent reporting, and learning-driven product decisions.

Qualifications

• PhD in a quantitative field with 5+ years applying it to real products or research.
• Proven track record in AI evaluation or measurement for products shipped to users.
• Strong communication skills to present to engineers, founders, and researchers.

Responsibilities

• Design evaluation frameworks and determine when a new model checkpoint is better.
• Build and maintain an internal benchmark for multi-turn tutoring.
• Link product data to model training using the Learn Engine.
• Define instrumentation and telemetry to capture required events.
• Publish parts of the benchmark to researchers outside the company.

Skills

Quantitative evaluation

Communication

Experiment design

Education

PhD in statistics / ML / CS / economics / physics

Tools

Python (Pandas/NumPy/SciPy/Jupyter)

SQL

MongoDB

PostgreSQL

GCP

Job description

Socket.dev in Beverly Hills is seeking a principal AI evaluation scientist to own the measurement layer under model training and learning science across StudyFetch and Honen. You will design evals, run benchmarks, and ensure safety and educational quality.

This founding-team role offers direct collaboration with decision-makers, end-to-end analytics ownership, and a culture of rigorous causality, transparent reporting, and learning-driven product decisions.