Staff / Principal Data Scientist

Metrosion Inc.; Burlingame (CA); On-site; USD 180,000 - 320,000; Full time

Job summary

Metrosion Inc. in Burlingame, California seeks a Staff or Principal Data Scientist to lead evaluation strategy for AI systems used in production workflows.

You will design experiments, measure model quality, latency, cost, and uncertainty, and guide product decisions with rigorous evidence. You will partner with engineers and domain experts to set evaluation standards, mentor colleagues, and translate findings into actionable product requirements while working on-site five days a week in our

Qualifications

• Staff level: at least 8 years of relevant professional experience in applied data science, machine learning, or quantitative research.
• Principal level: at least 12 years of relevant professional experience in applied data science, machine learning, or quantitative research.
• Prior professional experience working at an AI-forward company is required.
• A bachelor’s degree or higher in Computer Science, Data Science, Statistics, Mathematics, Artificial Intelligence, Machine Learning, or a closely related quantitative field from an accredited or officially recognized institution.
• A record of independently leading significant AI or ML work, setting evaluation standards, and guiding technical decisions with clear evidence.
• Strong programming skills in Python and Go are required, alongside SQL proficiency, practical statistics, experimental design, and experience working with imperfect real-world datasets.
• Experience evaluating machine learning systems and interpreting uncertainty, bias, and variation in results.
• Strong applied understanding of LLMs, embeddings, retrieval, and the limits of model-based evaluation.
• Ability to communicate findings and tradeoffs clearly to engineering, product, and domain stakeholders.

Responsibilities

• Design representative evaluation datasets, labeling guidance, and repeatable experiments for AI-assisted workflows.
• Measure factual quality, retrieval relevance, task completion, uncertainty, and performance across important user and data segments.
• Compare models and system configurations using quality, latency, inference cost, and human review effort.
• Investigate failure patterns and translate findings into product changes, better datasets, and release criteria.
• Document data provenance, evaluation limitations, and reproducibility while protecting sensitive information.

Skills

Python

Go

SQL

Statistics

Experimental design

LLMs

Retrieval

Education

Bachelor’s degree or higher in Computer Science, Data Science, Statistics, Artificial Intelligence, Machine Learning, or related field

Job description

Lead data science and AI evaluation at Staff or Principal level, making performance measurable, reproducible, and useful in real operational workflows.

The opportunity

Turn ambiguous quality questions into evidence that guides product decisions. Partner with engineers and domain experts to evaluate model behavior, improve retrieval and prediction, and understand the cost of errors in sensitive or consequential work.

This opening considers candidates at Staff or Principal level. Staff Data Scientists lead evaluation strategy and substantial applied research across product and engineering teams, establish rigorous methods, and mentor colleagues. Principal Data Scientists shape scientific direction across products, guide high-impact methodological decisions, and establish standards for evidence and model evaluation. Experience minimums apply to the level under consideration; demonstrated scientific leadership, scope, and impact also determine level.

What you will do

• Design representative evaluation datasets, labeling guidance, and repeatable experiments for AI-assisted workflows.
• Measure factual quality, retrieval relevance, task completion, uncertainty, and performance across important user and data segments.
• Compare models and system configurations using quality, latency, inference cost, and human review effort.
• Investigate failure patterns and translate findings into product changes, better datasets, and release criteria.
• Document data provenance, evaluation limitations, and reproducibility while protecting sensitive information.

What you bring

• Staff level: at least 8 years of relevant professional experience in applied data science, machine learning, or quantitative research.
• Principal level: at least 12 years of relevant professional experience in applied data science, machine learning, or quantitative research.
• Prior professional experience working at an AI-forward company is required.
• A bachelor’s degree or higher in Computer Science, Data Science, Statistics, Mathematics, Artificial Intelligence, Machine Learning, or a closely related quantitative field from an accredited or officially recognized institution.
• A record of independently leading significant AI or ML work, setting evaluation standards, and guiding technical decisions with clear evidence.
• Strong programming skills in Python and Go are required, alongside SQL proficiency, practical statistics, experimental design, and experience working with imperfect real-world datasets.
• Experience evaluating machine learning systems and interpreting uncertainty, bias, and variation in results.
• Strong applied understanding of LLMs, embeddings, retrieval, and the limits of model-based evaluation.
• Ability to communicate findings and tradeoffs clearly to engineering, product, and domain stakeholders.

Helpful experience

• Experience with retrieval evaluation, human labeling programs, drift monitoring, or production model monitoring.
• Work with privacy-sensitive data or operational decisions that require defensible evidence.
• A master’s degree or PhD in a relevant technical or quantitative discipline.

Location & working arrangement

This is a full-time position based in our Burlingame, California office and requires working on-site five days a week.

AI & ML knowledge

Robust theoretical and applied knowledge of AI and machine learning, including supervised and unsupervised learning, deep learning, generalization, calibration, sampling bias, and leakage-resistant validation.

Every team member should be able to discuss the capabilities and limitations of AI relevant to their work, verify AI-generated output, protect sensitive information, and keep their knowledge current as the technology evolves.

Before you apply

Direct applicants only

We work directly with individual applicants. We do not work with recruiting agencies, consulting firms, offshore or outsourcing companies, or freelancers offering external services for these openings. Please do not submit candidates or service proposals on behalf of a third party. Individuals may apply directly for our full-time positions.

All applicants must be legally authorized to work in the United States without employer sponsorship. Metrosion does not provide employment visa sponsorship.

All positions require working on-site in our Burlingame, California office five days a week. These roles are not remote or hybrid.

Different perspectives. Shared purpose.

We welcome applicants from all backgrounds and value an inclusive, respectful workplace.