Senior Data Scientist, AI Evaluation — Define & Validate Quality

Alpaca; New York (NY); On-site; USD 150,000 - 230,000; Full time

Benefits offered by this job

Competitive Salary & Stock Options

New Hire Home-Office Setup: USD 500

Monthly Stipend: USD 150

Job summary

Alpaca is hiring a Senior Data Scientist, AI Evaluation to define and measure how our models and agents are judged for correctness. You will craft ground truth, metrics, and evaluation loops, raising quality and speeding safe rollout on our broker-infrastructure platform.

You will partner with Product, Engineering, Analytics Engineering, and business stakeholders to define what good looks like and drive the evals that inform iteration.

Qualifications

• Strong statistical and ML foundation; treat evaluations as experiments with proper methodology.
• Experience evaluating models in production environments and defining quality metrics.
• Proficiency in Python and SQL; ability to evaluate models in production.

Responsibilities

• Design AI evaluations: define ground truth, metrics, and scoring methods.
• Build repeatable eval loops to track quality over time and catch regressions.
• Collaborate with engineering to operationalize eval harnesses and test infrastructure.
• Translate eval results into actionable improvements for systems.
• Establish and document evaluation standards and best practices.
• Mentor teams to adopt measurable AI quality practices.

Skills

Python

SQL

Education

Quantitative degree

Job description

Alpaca is hiring a Senior Data Scientist, AI Evaluation to define and measure how our models and agents are judged for correctness. You will craft ground truth, metrics, and evaluation loops, raising quality and speeding safe rollout on our broker-infrastructure platform.

You will partner with Product, Engineering, Analytics Engineering, and business stakeholders to define what good looks like and drive the evals that inform iteration.