Senior Data Scientist AI Evaluation

Alpaca; United States; Remote; USD 120,000 - 190,000; Full time

Job summary

Alpaca is seeking a Senior Data Scientist for AI Evaluation to design metrics and evaluation frameworks for models in a fintech setting. The role emphasizes a strong statistical background and quantitative measurement to ensure model quality before deployment.

You will build repeatable evaluation loops to monitor performance over time and collaborate with engineering and analytics teams to translate results into actionable system improvements.

Qualifications

• Strong statistical background.
• Experience designing metrics and evaluation frameworks.
• Ability to design repeatable evaluation loops and monitor model performance over time.

Responsibilities

• Design metrics and evaluation frameworks for AI models in fintech.
• Build repeatable evaluation loops to monitor performance and catch regressions.
• Collaborate with engineering and analytics teams to operationalize evaluation harnesses.

Skills

Statistical methods

Quantitative analysis

Metrics design

Education

Master's degree in Statistics/ML

Job description

Alpaca seeks a Senior Data Scientist for AI Evaluation to design metrics and evaluations for models in a fintech setting. Ideal candidates have a strong statistical background and experience in quantitative measurement.

In this role, you will focus on designing quality metrics and evaluation frameworks for AI models, ensuring they meet high standards before deployment. You will build repeatable evaluation loops to monitor model performance over time, catching regressions early in the process.

Collaboration is key; you will work alongside engineering and analytics teams to operationalize evaluation harnesses and translate results into actionable system improvements. Your expertise in statistical methods and quantitative measurement will guide the establishment of quality standards and best practices across the organization.