Senior AI Evaluation Scientist — Shape Safe Model Quality

Alpaca; United States; Remote; USD 140,000 - 230,000; Full time

Benefits offered by this job

Competitive Salary & Stock Options

New Hire Home-Office Setup: One-time 0

Monthly Stipend: USD $150 per month

Job summary

Alpaca, a US-headquartered leader in broker-developer infrastructure, is seeking a Senior Data Scientist, AI Evaluation to define how our models are evaluated and trusted. You will design ground truth, metrics, and evaluation loops, owning quality across teams and accelerating safe rollout.

You'll work with Product, Engineering, Analytics Engineering, and business stakeholders to codify what 'good' looks like, build rigorous evals, and ensure results drive iteration.

Qualifications

• 6–10 years in quantitative data science or ML, with focus on measurement or evaluation.
• Strong statistical foundation; treat evaluations as experiments with proper design.
• Proficiency in Python and SQL; experienced in evaluating models in production.

Responsibilities

• Design AI evaluations: Define ground truth, metrics, and scoring methods for models and agents.
• Build repeatable eval loops to track quality over time and catch regressions before release.
• Partner on infrastructure with engineering and analytics engineering to operationalize eval harnesses.
• Drive iteration: Translate eval results into actionable recommendations for system improvements.
• Establish evaluation guidelines, documentation, and review practices.
• Mentor and align: foster evaluation best practices and measurable AI quality across the team.

Skills

Quantitative data science

Model evaluation

Python

SQL

Cross-functional collaboration

Communication

Measurement

Education

Quantitative degree

Job description

Alpaca, a US-headquartered leader in broker-developer infrastructure, is seeking a Senior Data Scientist, AI Evaluation to define how our models are evaluated and trusted. You will design ground truth, metrics, and evaluation loops, owning quality across teams and accelerating safe rollout.

You'll work with Product, Engineering, Analytics Engineering, and business stakeholders to codify what 'good' looks like, build rigorous evals, and ensure results drive iteration.