Staff Data Scientist, AI Evaluation

Syllo

United States

Remote

USD 120,000 - 165,000

Full time

2 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Syllo is seeking a hands-on data scientist to design rigorous evaluation methods for AI system quality and performance using production data. You will run experiments, monitor for regressions, and translate results into actionable product and engineering improvements.

You will collaborate with Engineering, Product, and Analytics to communicate findings and guide system enhancements, focusing on measurable quality and scalable analytics solutions.

Qualifications

  • Strong data science skills for evaluating complex systems and interpreting results.
  • Experience designing experiments and quantitative measures of quality or performance.
  • Ability to work with production data, including extracting, transforming, and analyzing it.
  • Skill in investigating problems across data, models, and deployed systems.
  • Clear communication and ability to collaborate across technical and product teams.

Responsibilities

  • Create quantitative evaluation methods and tools for assessing AI system quality and performance.
  • Design and run experiments that compare approaches, reveal tradeoffs, and measure the effects of changes.
  • Establish repeatable monitoring to detect regressions, anomalies, and shifts in system performance.
  • Extract, transform, and analyze production data for technical and product investigations.
  • Diagnose issues involving data, models, and production systems, then identify likely causes and possible remedies.
  • Partner with Engineering, Product, and Analytics to communicate findings and guide improvements.

Skills

Data science skills
Experiment design
Production data wrangling
Investigating data/models
Communication & collaboration

Job description

Role overview

Develop rigorous ways to measure and improve the quality and performance of AI systems. This hands-on data science role spans applied AI, engineering, and product analytics: you will work with production data, design experiments, investigate system behavior, and turn findings into decisions and improvements.


Responsibilities


  • Create quantitative evaluation methods and tools for assessing AI system quality and performance.

  • Design and run experiments that compare approaches, reveal tradeoffs, and measure the effects of changes.

  • Establish repeatable monitoring to detect regressions, anomalies, and shifts in system performance.

  • Extract, transform, and analyze production data for technical and product investigations.

  • Diagnose issues involving data, models, and production systems, then identify likely causes and possible remedies.

  • Partner with Engineering, Product, and Analytics to communicate findings and guide improvements.


Requirements


  • Strong data science skills for evaluating complex systems and interpreting results.

  • Experience designing experiments and quantitative measures of quality or performance.

  • Ability to work with production data, including extracting, transforming, and analyzing it.

  • Skill in investigating problems across data, models, and deployed systems.

  • Clear communication and ability to collaborate across technical and product teams.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Quality & Evaluation Lead
AI Quality & Evaluation Lead

Millwonder • United States

Remote
USD 90,000 - 130,000
Remote, global work setup
Data Analyst (AI Evaluation)
Data Analyst (AI Evaluation)

micro1 Inc. • United States

Remote
USD 55,000 - 96,000
Applied Scientist / Research Engineer — Data Quality & Evaluation
Applied Scientist / Research Engineer — Data Quality & Evaluation

Maxinsights • Santa Clara (CA)

On-site
USD 140,000 - 210,000
Health insurance
Vision care
Dental coverage
+4
QA Engineer
QA Engineer

Ai Makers • United States

Remote
USD 120,000 - 170,000
Senior Director, Technical Product Management — AI Quality & Evaluation
Senior Director, Technical Product Management — AI Quality & Evaluation

Salesforce, Inc. • United States

Remote
USD 180,000 - 260,000
Medical coverage
Dental coverage
Vision coverage
+3
Data Scientist (AI Quality & Evaluation)
Data Scientist (AI Quality & Evaluation)

Bioscope.ai, Inc. • Boston (MA)

On-site
USD 100,000 - 130,000
Senior Data Scientist - AI
Senior Data Scientist - AI

CVS Health • United States

Remote
USD 120,000 - 170,000
Medical, dental, vision coverage
Paid time off
Retirement savings plan
+2
Staff Data Scientist: AI System Quality & Experiments
Staff Data Scientist: AI System Quality & Experiments

Syllo • United States

Remote
USD 120,000 - 165,000
AI Data Scientist
AI Data Scientist

Photon • Dallas (TX)

On-site
USD 180,000 - 260,000
Sr. Manager, AI Innovation.
Sr. Manager, AI Innovation.

Pinnacle Systems • United States

Remote
USD 140,000 - 210,000