Senior Data Platform Architect: Scale Pipelines & Observability

Perplexity AI Inc.; New York (NY); On-site; USD 130,000 - 180,000; Full time

Job summary

Perplexity AI Inc. is seeking a Senior/Staff Software Engineer for the Data Platform team. This role involves shaping architecture and driving the technical direction of the data ecosystem. You'll design and operate large-scale data pipelines and lead the architecture using tools like Airflow or Dagster.

Ideal candidates will have over 5 years of software engineering experience, strong production data infrastructure skills, and experience with batch and streaming data processing. If you're interested, please apply even if you don't match every qualification.

Qualifications

• 5+ years (Senior) or 8+ years (Staff) of software engineering experience.
• Strong experience building production data infrastructure systems.
• Hands-on experience with batch and/or streaming data processing at scale.
• Deep familiarity with data orchestration systems (Airflow, Dagster, or similar).
• Proficiency in Python and at least one additional backend language (Go, TypeScript, etc.).
• Strong systems thinking around reliability, latency, cost, and complexity tradeoffs.
• Experience supporting ML/AI workflows, training pipelines, or evaluation systems.
• Familiarity with data quality, lineage, observability, and governance tooling.

Responsibilities

• Design and operate large-scale batch and streaming data pipelines.
• Build event-driven and streaming systems for real-time ingestion and transformation.
• Lead the architecture of data orchestration using tools like Airflow or Dagster.
• Set and enforce guarantees for data correctness, freshness, lineage.
• Build self-serve data platforms for engineers and analysts.
• Drive architectural decisions across storage, compute, orchestration, and data APIs.
• Mentor engineers and raise technical standards for data infrastructure.

Skills

Software engineering experience

Production data infrastructure

Batch and streaming data processing

Data orchestration systems

Proficiency in Python

Systems thinking

Supporting ML/AI workflows

Data quality tooling

Tools

Airflow

Dagster

Kafka

Kinesis

PubSub

Job description

Perplexity AI Inc. is seeking a Senior/Staff Software Engineer for the Data Platform team. This role involves shaping architecture and driving the technical direction of the data ecosystem. You'll design and operate large-scale data pipelines and lead the architecture using tools like Airflow or Dagster.

Ideal candidates will have over 5 years of software engineering experience, strong production data infrastructure skills, and experience with batch and streaming data processing. If you're interested, please apply even if you don't match every qualification.