Senior Data Infrastructure Engineer - Real-Time Pipelines

Decagon; San Francisco (CA); On-site; USD 200,000 - 400,000; Full time

Benefits offered by this job

Medical, Dental, and Vision benefits

Retirement Plan (401K)

Parental Leave

Fertility and family building benefits

Wellness stipend

Daily lunches and snacks

Take what you need vacation policy

Job summary

Decagon is hiring a Senior Data Infrastructure Engineer to design, build, and operate the data systems powering our AI products. You will own critical data pipelines and storage end-to-end, improving reliability and performance for Decagon’s platforms.

You will lead infrastructure-as-code, observability, and latency optimizations while collaborating with research and product teams to scale features. This is an in-office role based in the United States.

Qualifications

• 5+ years building and operating production data infrastructure at scale.
• Hands-on experience with Tier 1 data technologies: ClickHouse, Kafka (or MSK/Pub-Sub/RabbitMQ), and Flink or dbt.
• Proven track record meeting high availability and low latency targets across streaming and batch workloads.
• Excellent observability chops (OpenTelemetry, Prometheus/Grafana, Datadog) and strong incident response discipline.
• Clear written communication and the ability to turn ambiguous data requirements into simple, reliable designs.

Responsibilities

• Design and implement high-throughput data pipelines and streaming systems with strong SLOs, clear runbooks, and actionable telemetry.
• Build and operate real-time and batch ingestion infrastructure using tools like Kafka, Flink, and Airflow.
• Own our analytical data layer — schema design, query performance, and cost optimization across ClickHouse, BigQuery, or similar.
• Partner with research and product teams to architect data solutions, evaluate performance, and scale new features.
• Tune pipeline and query latencies: optimize data paths, apply smart caching/partitioning, and hit tight p95/p99 targets.
• Lead infrastructure-as-code (Terraform) and GitOps practices for data systems; reduce drift with reusable modules and policy-as-code.
• Participate in on-call and drive down toil through automation and elimination of recurring data issues.

Skills

Data infrastructure

Streaming data

Observability

Incident response

Tools

Kafka

Flink

Airflow

dbt

ClickHouse

Job description

Decagon is hiring a Senior Data Infrastructure Engineer to design, build, and operate the data systems powering our AI products. You will own critical data pipelines and storage end-to-end, improving reliability and performance for Decagon’s platforms.

You will lead infrastructure-as-code, observability, and latency optimizations while collaborating with research and product teams to scale features. This is an in-office role based in the United States.