Principal Data Scientist, AI Evaluation & Product Impact

Metrosion Inc.; Burlingame (CA); Presencial; USD 180.000 - 320.000; Jornada completa

Descripción de la vacante

Metrosion Inc. in Burlingame, California seeks a Staff or Principal Data Scientist to lead evaluation strategy for AI systems used in production workflows.

You will design experiments, measure model quality, latency, cost, and uncertainty, and guide product decisions with rigorous evidence. You will partner with engineers and domain experts to set evaluation standards, mentor colleagues, and translate findings into actionable product requirements while working on-site five days a week in our

Formación

• Staff level: at least 8 years of relevant professional experience in applied data science, machine learning, or quantitative research.
• Principal level: at least 12 years of relevant professional experience in applied data science, machine learning, or quantitative research.
• Prior professional experience working at an AI-forward company is required.
• A bachelor’s degree or higher in Computer Science, Data Science, Statistics, Mathematics, Artificial Intelligence, Machine Learning, or a closely related quantitative field from an accredited or officially recognized institution.
• A record of independently leading significant AI or ML work, setting evaluation standards, and guiding technical decisions with clear evidence.
• Strong programming skills in Python and Go are required, alongside SQL proficiency, practical statistics, experimental design, and experience working with imperfect real-world datasets.
• Experience evaluating machine learning systems and interpreting uncertainty, bias, and variation in results.
• Strong applied understanding of LLMs, embeddings, retrieval, and the limits of model-based evaluation.
• Ability to communicate findings and tradeoffs clearly to engineering, product, and domain stakeholders.

Responsabilidades

• Design representative evaluation datasets, labeling guidance, and repeatable experiments for AI-assisted workflows.
• Measure factual quality, retrieval relevance, task completion, uncertainty, and performance across important user and data segments.
• Compare models and system configurations using quality, latency, inference cost, and human review effort.
• Investigate failure patterns and translate findings into product changes, better datasets, and release criteria.
• Document data provenance, evaluation limitations, and reproducibility while protecting sensitive information.

Conocimientos

Python

Go

SQL

Statistics

Experimental design

LLMs

Retrieval

Educación

Bachelor’s degree or higher in Computer Science, Data Science, Statistics, Artificial Intelligence, Machine Learning, or related field

Descripción del empleo

Metrosion Inc. in Burlingame, California seeks a Staff or Principal Data Scientist to lead evaluation strategy for AI systems used in production workflows.

You will design experiments, measure model quality, latency, cost, and uncertainty, and guide product decisions with rigorous evidence. You will partner with engineers and domain experts to set evaluation standards, mentor colleagues, and translate findings into actionable product requirements while working on-site five days a week in our