Senior Principal Data Engineer

Capital Factory; United States; On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Flexible paid time off

Medical, dental, and vision insurance

401(k) retirement plan with company 50

Employee Stock Purchase Plan

Educational assistance

Wellbeing benefits

Calm app subscriptions

Job summary

Cardlytics in the United States is seeking a Senior Principal Data Engineer to lead the design and implementation of our MLOps and forecasting platform on the Databricks Data Intelligence Platform. You will bridge data science and production engineering, ensuring scalable models for campaign projections and budget forecasting.

Collaborate with scientists, engineers, and product teams to establish CI/CD standards and robust pipelines, while mentoring staff and driving real-time data consumption.

Qualifications

• Bachelor's or Master's degree in CS/engineering or related field; PhD preferred.
• 10+ years of data engineering experience with MLOps pipelines in production.
• Expertise with Databricks Data Intelligence Platform (Unity Catalog, MLflow).
• Proficiency in Python, SQL, PySpark and large-scale analytics modeling.
• Track record leading large architectural migrations and managing technical debt.
• Strong knowledge of time-series forecasting, statistical modeling, or causal inference.

Responsibilities

• Design and architect a unified MLOps and forecasting platform on Databricks using MLflow and Databricks SQL.
• Build and maintain automated pipelines for campaign performance forecasting and budget utilization monitoring.
• Establish CI/CD and deployment standards for scalable, refreshed models.
• Define telemetry-first data standards and analyze large datasets for actionable insights.
• Mentor senior and staff engineers to improve operational velocity and engineering rigor.

Skills

Python

SQL

PySpark

Time-series forecasting

MLOps

Data modeling

Excellent communication

Education

Bachelor's or Master's degree in Computer Science or related field; PhD preferred

Tools

Databricks Data Intelligence Platform

Unity Catalog

MLflow

Databricks SQL

Kafka

Kinesis

CI/CD pipelines

Job description

About Cardlytics
Founded in 2008, Cardlytics (NASDAQ: CDLX) is the industry-leading purchase intelligence and incentives platform. We make commerce smarter and more rewarding for everyone by helping businesses attract, understand, and incentivize consumers through our partners' digital reward programs. Join us on our mission to make commerce smarter and more rewarding for everyone!

About the Team

The Ads Marketplace team at Cardlytics comprises a dynamic group of scientists and engineers dedicated to rethinking and redefining ad delivery and optimization at scale. Our work directly impacts millions of customers daily, driving innovation and effectiveness in how advertising reaches its audience. We are undergoing a strategic shift to unify our data engineering, experiment tracking, and model deployment within a robust Databricks-native ecosystem to accelerate time-to-market and enhance platform stability.

About the Position

As a Senior Principal Data Engineer, you will serve as the primary architect and strategic lead for our MLOps and forecasting platform. You will bridge the gap between advanced data science and production-grade engineering, ensuring that complex models for campaign projections, lift analysis, and budget forecasting are scalable, high-performance production services. You will lead the design of our data infrastructure, overseeing the implementation of automated pipelines using the Databricks Data Intelligence Platform (including Databricks SQL, Unity Catalog, and MLflow) to enable precise, real-time data consumption and model deployment.

You Will

• Design and architect a unified MLOps and forecasting platform on Databricks, leveraging MLflow for model orchestration and Databricks SQL for high-scale data processing.

• Build and maintain automated pipelines to handle critical business use cases, specifically:

  • Developing models that simulate past campaign performance using historical transaction data to forecast future outcomes.

  • Real-time monitoring and prediction of budget utilization to prevent over/under delivery.

  • Implementing robust Incrementality Testing framework to quantify the causal impact of advertising exposure.

• Establish CI/CD and deployment standards to ensure models are scalable, frequently refreshed, and easily debugged.

• Define "clean data" and telemetry-first standards. Analyze large datasets to identify trends and insights that improve model performance and advertising outcomes.

• Partner with Applied Scientists to explore GenAI patterns and automated architectures to enhance team workflows.

• Collaborate with product and business stakeholders to translate high-level business goals into technical roadmaps that prioritize explainable, scalable, and accurate engineering solutions.

• Act as a technical mentor for senior and staff engineers, fostering a culture of operational excellence and rapid dev velocity.

You Have

• Bachelor\'s or Master\'s degree in Computer Science, Engineering, or a related field; PhD preferred.

• 10+ years of experience in data engineering, with a significant focus on building and scaling MLOps pipelines in a production environment.

• Expertise in the Databricks Data Intelligence Platform, including Unity Catalog and MLflow, for big data and ML workloads.

• Proficiency in Python, SQL, and PySpark, with deep knowledge of data modeling and semantic layer design for large-scale analytics.

• A proven track record of leading large-scale architectural migrations and managing technical debt.

• Strong understanding of time-series forecasting, statistical modeling, or causal inference (e.g., Lift Analysis).

• Exceptional communication skills, with the ability to influence executive stakeholders and align technical strategy with business goals.

We Prefer That You Have

• Expertise in integrating complex prediction models (ranking, retrieval, and conversion) into high-traffic Marketplace funnels.

• Experience with real-time systems and large-scale data processing technologies (e.g., Kafka, Kinesis).

• Familiarity with Learning to Rank (LTR) models and their productionization challenges.

• Demonstrated ability to implement AI-powered assistants for product development, alert triage, and workload migration.

Technical Environment

We primarily use macOS and Google Workspace (Docs, Sheets, Slides). While our preferred platform is macOS, we support both macOS and Windows. Familiarity with or willingness to work within this environment is required.

Core Values

Our shared values are the driving force behind everything we do. Across all roles, we are looking for teammates who embody these values:

• Customer and partner first

• Act with urgency and focus

• Integrity with our partners and data

• Accountability even when challenged

• Empowerment over hierarchy

• Growth over comfort

Benefits and Perks

• Flexible paid time off plus company holidays

• Medical, dental, and vision insurance begins on your first day

• 401(k) retirement plan with company match, plan also includes a student loan debt repayment option

• Employee Stock Purchase Plan

• Educational assistance for continuing education

• Lifestyle Spending Account for physical, emotional, and financial wellness (like gym memberships, home down payments, art classes, park passes, and more!)

• Complimentary Calm app subscriptions to support employee mental health and wellbeing

As an equal opportunity employer, Cardlytics is committed to diversity, equity, and inclusion. Our people bring our products and organization to life, and every unique perspective makes us better. If you can do the job and you’re excited about growing with us as we scale our best‑in‑class advertising platform, we’d love to hear from you. If you need accommodation in the recruiting process due to a disability, please email [e-mail] or inform your recruiter.