Senior Data Scientist - Experimentation & Causal Inference

ByLabs; Seattle (WA); On-site; USD 150,000 - 230,000; Full time

Job summary

ByLabs in Seattle seeks a Senior Data Scientist specializing in Experimentation & Causal Inference to own the statistical framework of our experimentation program. You will design, validate, and improve methods that ensure trustworthy interpretations of results, while partners with product and platform teams for production-ready implementations.

You will leverage CUPED, SRM checks, and variance-reduction techniques to accelerate insights, with emphasis on AI-assisted tooling and

Qualifications

• MS or PhD in Statistics, Biostatistics, Econometrics, CS, or related quantitative field.
• 3+ years designing online controlled experiments at a tech company with meaningful user scale.
• Solid foundations in hypothesis testing, power analysis, multiple testing correction, and sequential testing.

Responsibilities

• Design and maintain automated anomaly detection for live experiments (SRM checks, traffic split validation).
• Define alert thresholds and circuit-breaking criteria to protect experiments.
• Define and validate guardrail metrics for experiment health and interpretability.
• Implement CUPED and covariate adjustment to reduce variance and improve detection power.
• Translate methods into production-ready specifications, validation scripts, and self-serve tooling.

Skills

Statistical analysis

Experiment design

Communication

Bilingual communication English/中文 (英语

AI tooling proficiency

Education

MS or PhD in Statistics / Econometrics / related quantitative field

Tools

Python

SQL

Claude Code/OpenClaw

Job description

We are looking for a Senior Data Scientist - Experimentation & Causal Inference to serve as the statistical brain behind our rapidly growing experimentation program. You will own the methodology layer that ensures every experiment we run is trustworthy, sensitive, and correctly interpreted.

You will NOT be building infrastructure or writing backend services — our platform engineering team handles that. Your job is to design, validate, and continuously improve the statistical frameworks that sit on top of the platform. You will be the person the team turns to when they ask "Can we trust this result?" — and you will build the automated systems that answer that question before anyone needs to ask.

Team context: You will join a team of data scientists and platform engineers within the Big Data group, reporting to the Head of Big Data. The experimentation platform engineering team implements your specifications into production systems.

Responsibility

• Design and maintain automated anomaly detection for live experiments — including Sample Ratio Mismatch (SRM) checks and traffic split validation.
• Define alerting thresholds and circuit-breaking criteria so compromised experiments are flagged or stopped before polluting decisions.
• Define and validate guardrail metrics (sensitivity, directionality, interpretability) that protect the business during every experiment.

Variance Reduction & Sensitivity

• Implement and iterate on CUPED and related pre-experiment covariate adjustment methods to reduce metric variance.
• Develop techniques to remove noise from user historical behavior, enabling faster detection of true treatment effects — especially in limited-traffic or high-priority scenarios.
• Example of impact we are targeting: Shorten average experiment duration from 14 days to 9 days, unlocking 40%+ more experiments per quarter.
• Design and run continuous A/A experiments as an always-on health check for the data pipeline (from client-side event reporting through message queues to the real-time data warehouse).
• Monitor metric baseline volatility and pipeline stability, ensuring the instrumentation layer remains trustworthy over time. You define what "healthy" looks like; the platform team implements the monitoring infrastructure.

Other responsibilities

• Partner with product, engineering, and growth teams on experiment design: sample size calculations, metric selection, duration estimation, and result interpretation. Advise on causal inference methods (DID, synthetic control, RDD) when randomization is not feasible.
• Translate your methods into specifications, validation scripts, and decision frameworks that the team can operationalize — enabling experiment owners to self-serve while maintaining rigor.

Requirements

• Education: MS or PhD in Statistics, Biostatistics, Economics (Econometrics), Computer Science, or a related quantitative field.
• Experience: 3+ years of industry experience designing and analyzing online controlled experiments at a tech company with meaningful user scale (not exclusively survey experiments or clinical trials).
• Core Statistical Knowledge: Solid foundations in hypothesis testing, power analysis, multiple testing correction, and sequential testing. Hands-on experience with at least two of: variance reduction methods (CUPED or similar), sample ratio mismatch detection, or continuous data quality monitoring for experiments.
• Programming: Proficient in Python (scipy, statsmodels, or equivalent) for statistical analysis and simulation. Comfortable writing complex SQL (window functions, CTEs) for data extraction and validation.
• Communication & Collaboration: Ability to explain complex statistical concepts to non-technical stakeholders, translate business questions into rigorous experimental designs, and define clear specifications so engineers can implement your methods in production.
• AI-Native Workflow: Strong AI Sense — extensive hands-on experience with AI coding tools (Claude Code, OpenClaw, or similar). Demonstrated ability to leverage AI assistants to automate repetitive analytical tasks, accelerate code development, and build self-serve tooling faster. You treat AI tools as a daily productivity multiplier, not a novelty.
• Bilingual Communication: Fluent in both English and Chinese, able to collaborate effectively with Asia-Pacific engineering and product teams, serving as a communication bridge between the US R&D Center and APAC teams. This role involves occasional cross-timezone coordination with teams in Singapore, Dubai, and other APAC locations (UTC+4 to UTC+8); candidates may need to join important meetings in early mornings or evenings.

Nice-to-Haves

• Familiarity with Bayesian experimentation frameworks or multi-armed bandit approaches.
• Understanding of real-time streaming data pipelines (Kafka, Flink) — not as an engineer, but how data flow affects statistical validity.
• Publications or conference presentations on experimentation methodology or causal inference (KDD, WSDM, CIKM, NeurIPS CausalML workshop, or equivalent).
• Experience in fintech, crypto, or high-frequency trading domains where user behavior has high variance and fast-moving baselines.