Lead Data Engineer — Real-Time Pipelines & Experimentation

Eliassen Group; Santa Monica (CA); Hybrid; USD 114,000 - 128,000; Full time

Benefits offered by this job

Medical, Dental, Vision benefits

401k with company matching

Life insurance

Job summary

Eliassen Group is seeking a hands‑on Lead Data Engineer in Santa Monica, CA, with 7+ years building large‑scale data platforms. You will design and operate batch and real‑time pipelines using Spark, Kafka, and Databricks to power A/B tests and product analytics.

This hybrid role requires strong data modeling, governance, and collaboration with product and engineering teams. W2 role with a comprehensive benefits package including medical, dental, vision, 401k with company matching, and life

Qualifications

• 7+ years of data engineering with large-scale data platforms.
• Strong Python and SQL development.
• Expert-level Databricks and distributed processing experience.
• Deep knowledge of data modeling, ETL/ELT design, and data architecture.
• Experience with batch and real‑time streaming data pipelines, including Kafka and Spark Structured Streaming.
• Background in cloud-based data platforms and data warehousing or lakehouse technologies.
• Hands‑on experience with experimentation, A/B testing, product analytics, or analytics data platforms.
• Strong understanding of data quality, testing, governance, monitoring, and observability.

Responsibilities

• Design, develop, and maintain data infrastructure supporting A/B testing, feature launches, geo experiments, and product analytics.
• Build reusable frameworks and datasets to enable rapid, reliable experiment launch and measurement.
• Develop and operate large-scale batch and streaming pipelines using Spark, Kafka, and Databricks.
• Design scalable data models and data products for analytics, experimentation, and machine learning use cases.
• Implement monitoring, testing, validation, lineage, and governance frameworks to ensure trustworthy data.
• Partner with Product, Engineering, Data Science, and Business stakeholders to deliver scalable solutions.
• Influence architectural decisions and drive engineering best practices across teams.
• Mentor engineers and promote a culture of quality, innovation, and experimentation.

Skills

SQL

Python

Databricks

Data modeling

Streaming data

Kafka

Spark

Cloud platforms

A/B testing

Leadership

Tools

Databricks

Spark

Kafka

Job description

Eliassen Group is seeking a hands‑on Lead Data Engineer in Santa Monica, CA, with 7+ years building large‑scale data platforms. You will design and operate batch and real‑time pipelines using Spark, Kafka, and Databricks to power A/B tests and product analytics.

This hybrid role requires strong data modeling, governance, and collaboration with product and engineering teams. W2 role with a comprehensive benefits package including medical, dental, vision, 401k with company matching, and life