Senior Data Scientist — AI Evaluation & Quality Metrics

Alpaca; United States; Remote; USD 120,000 - 190,000; Full time

Job summary

Alpaca is seeking a Senior Data Scientist for AI Evaluation to design metrics and evaluation frameworks for models in a fintech setting. The role emphasizes a strong statistical background and quantitative measurement to ensure model quality before deployment.

You will build repeatable evaluation loops to monitor performance over time and collaborate with engineering and analytics teams to translate results into actionable system improvements.

Qualifications

• Strong statistical background.
• Experience designing metrics and evaluation frameworks.
• Ability to design repeatable evaluation loops and monitor model performance over time.

Responsibilities

• Design metrics and evaluation frameworks for AI models in fintech.
• Build repeatable evaluation loops to monitor performance and catch regressions.
• Collaborate with engineering and analytics teams to operationalize evaluation harnesses.

Skills

Statistical methods

Quantitative analysis

Metrics design

Education

Master's degree in Statistics/ML

Job description

Alpaca is seeking a Senior Data Scientist for AI Evaluation to design metrics and evaluation frameworks for models in a fintech setting. The role emphasizes a strong statistical background and quantitative measurement to ensure model quality before deployment.

You will build repeatable evaluation loops to monitor performance over time and collaborate with engineering and analytics teams to translate results into actionable system improvements.