Lead Data Engineer - Experimentation Platform

Jobtailor; Santa Monica (CA); On-site; USD 140,000 - 200,000; Full time

Job summary

Jobtailor is seeking a senior Data Engineer to design and build scalable data platforms, supporting experimentation and A/B testing. You will develop batch and streaming pipelines for large-scale user and product datasets.

You will create analytics-ready datasets, dimensional models, and data products while implementing data quality, governance, and CI/CD. Collaborating with product, engineering, data science, and analytics teams, you will mentor engineers and advance data infrastructure for ML

Qualifications

• Bachelor's degree in Computer Science, Engineering, Mathematics, Statistics, or a related technical field.
• 7+ years of experience in data engineering or large-scale data platforms.
• Strong experience with distributed data processing and cloud-based data architectures.
• Hands-on experience with Python, SQL, Spark, Databricks, Snowflake, Kafka, and Airflow.
• Strong understanding of data modeling, ETL/ELT, streaming architectures, and lakehouse concepts.
• Experience building experimentation, analytics, personalization, or ML data platforms.
• Experience implementing CI/CD, automated testing, monitoring, and data governance.
• Strong system design and architecture experience.
• Experience mentoring engineers and leading technical initiatives.

Responsibilities

• Design and build scalable data platforms supporting experimentation and A/B testing.
• Develop batch and streaming data pipelines for large-scale user and product datasets.
• Build reusable datasets and frameworks for experimentation, analytics, and product measurement.
• Design dimensional data models and analytics-ready data products.
• Implement automated data quality, validation, monitoring, lineage, and governance.
• Build production-grade deployment pipelines with CI/CD and observability.
• Partner with Product, Engineering, Data Science, and Analytics teams to deliver scalable data solutions.
• Optimize data infrastructure supporting experimentation, personalization, and machine learning workloads.
• Mentor engineers and establish best practices for large-scale data engineering.

Skills

Distributed data processing

Cloud-based data architectures

Mentoring engineers

Data governance

CI/CD for data pipelines

Education

Bachelor's degree in Computer Science or related field

Tools

Python

SQL

Spark

Databricks

Snowflake

Kafka

Airflow

Job description

Responsibilities

• Design and build scalable data platforms supporting experimentation and A/B testing.
• Develop batch and streaming data pipelines for large-scale user and product datasets.
• Build reusable datasets and frameworks for experimentation, analytics, and product measurement.
• Design dimensional data models and analytics-ready data products.
• Implement automated data quality, validation, monitoring, lineage, and governance.
• Build production-grade deployment pipelines with CI/CD and observability.
• Partner with Product, Engineering, Data Science, and Analytics teams to deliver scalable data solutions.
• Optimize data infrastructure supporting experimentation, personalization, and machine learning workloads.
• Mentor engineers and establish best practices for large-scale data engineering.

Requirements

• Bachelor's degree in Computer Science, Engineering, Mathematics, Statistics, or a related technical field.
• 7+ years of experience in data engineering or large-scale data platforms.
• Strong experience with distributed data processing and cloud-based data architectures.
• Hands-on experience with Python, SQL, Spark, Databricks, Snowflake, Kafka, and Airflow.
• Strong understanding of data modeling, ETL/ELT, streaming architectures, and lakehouse concepts.
• Experience building experimentation, analytics, personalization, or ML data platforms.
• Experience implementing CI/CD, automated testing, monitoring, and data governance.
• Strong system design and architecture experience.
• Experience mentoring engineers and leading technical initiatives.