Senior Data Scientist - Experimentation & Causal Inference

Confidential; San Francisco (CA); On-site; USD 120,000 - 150,000; Full time

Job summary

A Confidential company in San Francisco is seeking an experienced Experimentation Engineer to design and build the core experimentation platform infrastructure. You will work on automated metric pipelines, advanced statistical methods, and self-serve tooling while collaborating with cross-functional teams to analyze impactful experiments.

The ideal candidate will have 5+ years of experience in experimentation engineering, strong proficiency in Python, SQL, and bilingual skills in English and Mandarin for effective cross-timezone collaboration.

Qualifications

• 5+ years of industry experience in experimentation engineering, data engineering, or growth engineering.
• Proven track record building A/B testing infrastructure.
• Deep expertise in causal inference and experimental statistics.
• Strong proficiency in Python and SQL.
• Experience in building data products and self-serve analytics.
• Bilingual communication skills in English and Mandarin.
• Experience in fintech/crypto is a plus.

Responsibilities

• Design and build the core experimentation platform infrastructure.
• Implement advanced statistical methods for experiments.
• Build automated metric pipelines connecting experiments to business outcomes.
• Partner with cross-functional teams to analyze high-impact experiments.
• Establish experiment quality standards and monitoring.
• Mentor junior engineers and drive best practices.

Skills

Experimentation engineering

Causal inference

Python

SQL

Statistical analysis

Cross-device identity resolution

Real-time data processing

Bilingual in English and Mandarin

Tools

Flink Streaming

Spark Structured Streaming

Hive

ClickHouse

Kafka

Job description

Responsibilities

• Design and build the core experimentation platform infrastructure: experiment assignment service, randomization and traffic splitting, multi-layer experiment conflict detection, experiment configuration management (low-code, ≤30 min setup), and full lifecycle tooling (creation, monitoring, graduation, rollback) — supporting App, Web, and backend surfaces simultaneously
• Build automated metric pipelines that connect experiment assignments to full-funnel business outcomes — Signup CVR, eFTD CVR, eFTT CVR, deposit amount, trading volume — with sub-day latency; solve the cross-device identity bridging problem (device ID to user ID across the registration boundary) and implement "time-to-convert" metrics (e.g., eFTD within N minutes of page exposure) that are currently unavailable
• Implement advanced statistical methods: CUPED and stratified variance reduction to improve experiment sensitivity without increasing sample size, sequential testing for early stopping (mSPRT / always-valid inference), network interference correction for referral and social experiments, and pre-experiment SRM (Sample Ratio Mismatch) checks; design and execute conversion lift studies to quantify the causal impact of product changes on business metrics
• Build self-serve experiment creation and analysis tooling for product managers, growth marketers, and data scientists — including experiment design wizards, power calculators, automated significance reporting, and decision support dashboards; reduce the experiment launch cycle from "requires 2 days of engineering" to "self-serve in under 30 minutes" for all three platforms (App, Web, backend)
• Establish experiment quality standards: event instrumentation requirements, guardrail metric monitoring, automated anomaly detection to prevent shipping regressions, and an experiment knowledge base that documents winning patterns, failed hypotheses, and domain-specific learnings — ensuring teams learn from each other rather than rediscovering the same findings
• Partner with Growth Product, Personalization (千人千面), TradeGPT, ByX Community, and Asia-Pacific data engineering teams to design and analyze high-impact experiments across personalization, campaign optimization, user journey, push notifications, community feed, and AI product features; serve as the internal authority on causal inference and experiment design across all business units
• Build the experimentation platform as the feedback engine for Bybit's AI strategy: design automated attribution systems and causal inference pipelines that deliver real-time feedback signals to AI models (TradeGPT response ranking, personalization algorithms, push notification optimization) — enabling AI models to self-iterate based on causal experiment results rather than correlation-only metrics; build experiment bloodline tracking that traces how each AI model version performs across user segments, and end-to-end observability for recommendation and growth systems that accelerates the iteration cycle from research to production deployment
• Define engineering standards, conduct design reviews, and mentor junior engineers; drive cross-team adoption of experimentation best practices as the US team grows

Major Requirements

• 5+ years of industry experience in experimentation engineering, data engineering, or growth engineering at a consumer-scale internet company
• Proven track record building and operating large-scale A/B testing infrastructure: experiment assignment, metric pipelines, statistical analysis, and self-serve tooling serving hundreds of experiments simultaneously
• Deep expertise in causal inference and experimental statistics: hypothesis testing, power analysis, CUPED/variance reduction, sequential testing (mSPRT, always-valid inference), network effects and interference correction, conversion lift studies, and treatment effect estimation; ability to apply statistical test theories to optimize user experience and validate business decisions
• Strong proficiency in Python and SQL; hands‑on experience with real-time data processing frameworks — Flink Streaming or Spark Structured Streaming — as well as data warehouses (Hive, ClickHouse, or equivalent), real-time messaging (Kafka), and cross-device identity resolution (device ID to user ID mapping across the registration boundary)
• Experience building data products and self‑serve analytics tooling; strong sense for developer experience and product design for internal platforms; ability to reduce experiment launch cycles from days to minutes through thoughtful platform design
• Strong bilingual communication skills in both English and Mandarin Chinese; ability to collaborate effectively with Asia-Pacific engineering and product teams in Mandarin, bridging the US R&D Center with Asia-Pacific teams. This role involves cross‑timezone collaboration with teams in Singapore, Dubai, and other Asia-Pacific locations (UTC+8 to UTC+4); candidates may occasionally have important cross‑timezone meetings in the early morning or evening
• Nice-to-have: experience in fintech/crypto with understanding of financial user behavior and conversion funnels; experience with causal ML methods (DML, IV, synthetic control, uplift modeling) for observational analysis; experience with intelligent marketing or subsidy optimization experiments; familiarity with experiment platforms at Meta (PlanOut/Ax), Netflix, Airbnb (Experimentation Platform), or LinkedIn (XLNT); publications at top venues (KDD, NeurIPS, WWW, SIGIR, WSDM, CIKM, ICLR, ICML, or statistics journals); prior founding team or early-stage R&D center experience