Lead Engineer - Data Platform & AI Agents

The Briefed; Boston (MA); On-site; USD 180,000 - 260,000; Full time

Job summary

Klaviyo is seeking a Principal Data Platform Engineer to own the overall technical direction of its data platform, spanning streaming ingestion, batch compute, lakehouse storage, and governance. This IC role requires leading system design without direct reports and delivering fast, reliable, and cost-efficient data infrastructure.

You will mentor senior staff, drive paved roads for data producers and consumers, and collaborate across teams to ensure scalable, secure data access for analysts and

Qualifications

• 10+ years of professional experience building large-scale distributed data systems.
• Expertise across streaming and batch data stacks with multi-tenant reliability.
• Strong track record delivering data contracts, governance, and scalable platforms.

Responsibilities

• Core Data Platform Architecture: design and implement real-time ingestion, stream processing, and lakehouse storage.
• Define SLOs for data freshness, availability, and correctness; manage ADRs and post-incident reviews.
• Data Governance & Contracts: establish data contracts, schema governance, and reusable templates.
• Performance & FinOps: tune distributed compute for cost efficiency and reliability.
• AI-Augmented Data Ops: apply GenAI tools for semantic search, auto-docs, and AI-assisted quality checks.
• Mentorship: guide senior engineers through reviews and design doc oversight.

Skills

Kafka

Pub/Sub

Flink

Spark Streaming

Beam

ML/GenAI tooling

Tools

Airflow

Dagster

Spark

Iceberg

Delta Lake

Snowflake

BigQuery

S3/GCS

CDC pipelines

Job description

At Klaviyo, we empower creators to own their destiny by making first-party data accessible and actionable. Our data platform processes billions of events daily across billions of consumer profiles for hundreds of thousands of brands. Increasingly, this data powers autonomous AI agents acting on behalf of global enterprise customers.

As a Principal Data Platform Engineer, you will own the overall technical direction, system architecture, and delivery of Klaviyo’s data platform—spanning streaming ingestion, batch compute, storage/lakehouse, and schema governance. In this high-leverage individual contributor (IC) role, you will lead without direct reports by designing systems that are fast, reliable, cost-efficient, and secure for autonomous access. You will build standardized "paved roads" for data producers and consumers (human analysts and AI agents alike) while mentoring senior and staff engineers across the company.

Key Responsibilities

• Core Data Platform Architecture: Design and implement distributed data platform capabilities, including real-time event ingestion/CDC, stream processing, batch orchestration, lakehouse storage patterns, data catalogs, lineage, and access compliance.
• SLOs & Reliability Engineering: Define and uphold strict Service Level Objectives (SLOs) for data freshness (≥99.9%), availability, and correctness. Drive architecture decision records (ADRs), RFCs, readiness reviews, and post-incident learning.
• Data Governance & Contracts: Establish data contracts, schema governance frameworks, standardized connectors, and reusable pipeline templates to accelerate developer velocity across product teams.
• Performance & FinOps Optimization: Profile, tune, and right-size distributed compute clusters for performance and unit-economics cost efficiency in partnership with FinOps.
• AI-Augmented Data Operations: Apply GenAI and ML tools to streamline data workflows—enabling semantic catalog search, automated documentation, SQL/pipeline generation, and AI-assisted data quality anomaly detection under human-in-the-loop controls.
• Technical Mentorship & Enablement: Mentor Staff and Senior Engineers across pillars through architectural pairing, code reviews, and design doc oversight.

Requirements

Experience & Distributed Systems Mastery

• 10+ years of professional experience building and operating large-scale distributed data systems with multi-tenant reliability.
• Deep expertise across distributed data stack components:
  • Streaming & Messaging: Kafka, Pub/Sub, Flink, Spark Streaming, Beam.
  • Batch Orchestration & Engines: Airflow, Dagster, Spark.
  • Lakehouse & Warehouses: Apache Iceberg, Delta Lake, Hudi, Snowflake, BigQuery.
  • Storage & Ingestion: Object storage (S3/GCS), CDC pipelines, and schema registries.
• Proven track record establishing and meeting strict SLOs for data freshness, availability, and correctness.

AI Fluency & Influence

• Hands-on experience applying ML/GenAI tools to data infrastructure (semantic search, auto-docs, DQ monitors, AI-assisted pipeline generation).
• Demonstrated history of landing data contracts and standardized paved roads across cross-functional engineering teams.
• Exceptional technical writing and communication skills for authoring RFCs, ADRs, and system design specifications.

Preferred Qualifications & Success Indicators

• Experience with regional isolation, multi-region replication strategies, and privacy-by-design data governance in regulated environments.
• Prior track record achieving ≥99.9% data freshness SLOs and driving measurable cost/TB reductions across petabyte-scale environments.
• Experience reducing pipeline authoring times and data incidents through AI-in-the-loop operational tooling.