Staff Data Scientist: AI System Quality & Experiments

Syllo

United States

Remote

USD 120,000 - 165,000

Full time

2 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Syllo is seeking a hands-on data scientist to design rigorous evaluation methods for AI system quality and performance using production data. You will run experiments, monitor for regressions, and translate results into actionable product and engineering improvements.

You will collaborate with Engineering, Product, and Analytics to communicate findings and guide system enhancements, focusing on measurable quality and scalable analytics solutions.

Qualifications

  • Strong data science skills for evaluating complex systems and interpreting results.
  • Experience designing experiments and quantitative measures of quality or performance.
  • Ability to work with production data, including extracting, transforming, and analyzing it.
  • Skill in investigating problems across data, models, and deployed systems.
  • Clear communication and ability to collaborate across technical and product teams.

Responsibilities

  • Create quantitative evaluation methods and tools for assessing AI system quality and performance.
  • Design and run experiments that compare approaches, reveal tradeoffs, and measure the effects of changes.
  • Establish repeatable monitoring to detect regressions, anomalies, and shifts in system performance.
  • Extract, transform, and analyze production data for technical and product investigations.
  • Diagnose issues involving data, models, and production systems, then identify likely causes and possible remedies.
  • Partner with Engineering, Product, and Analytics to communicate findings and guide improvements.

Skills

Data science skills
Experiment design
Production data wrangling
Investigating data/models
Communication & collaboration

Job description

Syllo is seeking a hands-on data scientist to design rigorous evaluation methods for AI system quality and performance using production data. You will run experiments, monitor for regressions, and translate results into actionable product and engineering improvements.

You will collaborate with Engineering, Product, and Analytics to communicate findings and guide system enhancements, focusing on measurable quality and scalable analytics solutions.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff Data Scientist, AI Evaluation
Staff Data Scientist, AI Evaluation

Syllo • United States

Remote
USD 120,000 - 165,000
Senior Data Scientist — Remote AI Evaluation & Experimentation
Senior Data Scientist — Remote AI Evaluation & Experimentation

Synthires • San Francisco (CA)

Remote
USD 207,000 - 276,000
Fully remote work
Remote QA Automation Engineer: Build Reliable AI Tests
Remote QA Automation Engineer: Build Reliable AI Tests

Syllo • United States

Remote
USD 150,000 - 210,000
Health insurance
Equity
Applied Scientist: Data Quality & Evaluation
Applied Scientist: Data Quality & Evaluation

Maxinsights • Santa Clara (CA)

On-site
USD 140,000 - 210,000
Health insurance
Vision care
Dental coverage
+4
Remote Data Scientist - AI Model Evaluation & Experimentation
Remote Data Scientist - AI Model Evaluation & Experimentation

Synthires • United States

Remote
USD 207,000 - 276,000
Fully remote
Founding Data Scientist: AI Experimentation & Impact
Founding Data Scientist: AI Experimentation & Impact

present of AI • San Francisco (CA)

On-site
USD 120,000 - 160,000
Remote QA Automation Engineer - AI-Driven Test Framework
Remote QA Automation Engineer - AI-Driven Test Framework

Syllo • United States

Remote
USD 150,000 - 210,000
Health insurance
Equity
AI Analytics Lead: Experiments & Impact
AI Analytics Lead: Experiments & Impact

present of AI • San Francisco (CA)

On-site
USD 120,000 - 170,000
Staff Data Scientist: Agent Improvement & Quality Lead
Staff Data Scientist: Agent Improvement & Quality Lead

Yalo • United States

Remote
USD 180,000 - 260,000
Unlimited PTO
Remote work
Flexible hours
+3
Remote Data Scientist: AI Evaluation & Experimentation
Remote Data Scientist: AI Evaluation & Experimentation

Synthires • San Francisco (CA)

Remote
USD 207,000 - 276,000
Fully remote