Senior Data Scientist, AI Evaluation (Fully Remote)

Alpaca; United States; Remote; USD 180,000 - 240,000; Full time

Job summary

Alpaca, a global leader in brokerage infrastructure, seeks an experienced ML quality scientist to own the ground truth, metrics, and evaluation framework for our models and agents.

You will build repeatable eval loops, catch regressions before release, and collaborate with engineering and analytics to operationalize eval harnesses.

Qualifications

• 6–10 years of experience in quantitative data science or ML with measurement rigor.
• Strong Python, SQL, and statistical skills.
• LLM evaluation experience a plus.

Responsibilities

• Build repeatable eval loops and catch regressions before release.
• Partner with engineering and analytics to operationalize eval harnesses.

Skills

Quantitative data science

ML / machine learning

Python

SQL

Statistical skills

LLM evaluation experience

Job description

• Own the quality bar for Alpaca's models and agents.
• Define ground truth, metrics, and scoring methods for AI evaluations.

What You'll Do:

• Build repeatable eval loops and catch regressions before release.
• Partner with engineering and analytics to operationalize eval harnesses.

What We're Looking For:

• 6-10 years in quantitative data science or ML with measurement rigor.
• Strong Python, SQL, and statistical skills; LLM eval experience a plus.

Alpaca

Alpaca is a global leader in agent-first brokerage infrastructure, providing APIs for stocks, ETFs, options, crypto, and more. We\'re a diverse team of 400+ remote-first members backed by $400M in funding, committed to opening financial services to everyone.