Lead Data Platform Architect for AI & Real-Time Analytics

Maven Ventures; Palo Alto (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Perplexity is seeking a senior/staff Data Platform Engineer to shape architecture and set standards for its data ecosystem. You will lead large-scale batch and streaming pipelines powering product features, analytics, and AI workloads, and drive end-to-end observability and data quality.

You will mentor engineers and collaborate with product engineering and data science to align with the roadmap. You will build self-serve data platforms, improve developer experience, and define contracts and

Qualifications

• 5+ years (Senior) or 8+ years (Staff) of software engineering experience.
• Strong experience building production data infrastructure systems.
• Hands-on experience with batch and/or streaming data processing at scale.
• Deep familiarity with data orchestration systems (Airflow, Dagster, or similar).
• Proficiency in Python and at least one additional backend language (Go, TypeScript, etc.).
• Strong systems thinking around reliability, latency, cost, and complexity tradeoffs.
• Experience supporting ML/AI workflows, training pipelines, or evaluation systems.
• Familiarity with data quality, lineage, observability, and governance tooling.
• Prior ownership of internal platforms used by many teams.

Responsibilities

• Design and operate large-scale batch and streaming data pipelines powering product features, AI training and evaluation workflows, analytics, and experimentation.
• Build event-driven and streaming systems for real-time ingestion, transformation, and delivery, plus batch backfills and offline computation.
• Lead the architecture of data orchestration using Airflow or Dagster with end-to-end observability.
• Ensure data correctness, freshness, lineage, and recoverability amid rapid scale and evolving schemas.
• Build self-serve data platforms enabling engineers, data scientists, and analysts to discover data and create pipelines with minimal friction.
• Improve developer experience with abstractions, paved paths, and standards for data modeling, testing, validation, and deployment.
• Drive architectural decisions across storage, compute, orchestration, and data APIs in alignment with roadmap.
• Mentor engineers, review designs, and raise the technical bar with documentation and collaboration.

Skills

Python

Go/TypeScript

Batch processing

Streaming data

Data orchestration

Observability

Mentoring

ML/AI workflows

Tools

Airflow

Dagster

Kafka

Kinesis

PubSub

Spark

Iceberg

Delta Lake

Job description

Perplexity is seeking a senior/staff Data Platform Engineer to shape architecture and set standards for its data ecosystem. You will lead large-scale batch and streaming pipelines powering product features, analytics, and AI workloads, and drive end-to-end observability and data quality.

You will mentor engineers and collaborate with product engineering and data science to align with the roadmap. You will build self-serve data platforms, improve developer experience, and define contracts and