Senior Distributed Storage Engineer – Parquet & DataFusion

Datadog, Inc.; Boston (MA); Hybrid; USD 192,000 - 240,000; Full time

Benefits offered by this job

New hire stock equity (RSUs)

Employee stock purchase plan (ESPP)

Continuous professional development

Mentor and buddy program

Inclusive culture and mental health

Global benefits

401(k) plan and match

Vacation and wellness reimbursement

Job summary

Datadog is seeking a Senior Software Engineer to join Husky, our petabyte-scale distributed storage engine, to evolve the platform toward Parquet and DataFusion while supporting AI-driven workloads. You will collaborate closely with Husky Ingestion, Metadata, and Platform teams and contribute to open-source efforts.

You will work with teams across the stack to optimize storage, reduce costs, and enable scalable analytics, with a culture that values remote flexibility and strong ownership.

Qualifications

• Experience with large-scale distributed systems and storage engines.
• Strong fundamentals in columnar/file storage formats or comparable systems.
• Bonus: Go, Java, or Rust; compaction or compression schemes; object storage at scale; Kafka and stream processing; Kubernetes.

Responsibilities

• Migrate storage format to Parquet and re-imagine compaction on a DataFusion-based substrate.
• Drive cost-reduction work: block-level index compression, row-level object integration, storage tiering/caching.
• Scale the solution to manage event versioning at scale using temporal-database techniques.
• Operate and influence how to best run the service going forward.
• Collaborate across write path (Ingestion) and handoff to Lakehouse for batch/analytical workloads.

Skills

Distributed systems

Storage engines

Parquet/ORC/Arrow

DataFusion

Go/Java/Rust

S3/GCS/Azure Blob

Kafka

Kubernetes

Tools

Kafka

Kubernetes

S3/GCS/Azure Blob

Job description

Datadog is seeking a Senior Software Engineer to join Husky, our petabyte-scale distributed storage engine, to evolve the platform toward Parquet and DataFusion while supporting AI-driven workloads. You will collaborate closely with Husky Ingestion, Metadata, and Platform teams and contribute to open-source efforts.

You will work with teams across the stack to optimize storage, reduce costs, and enable scalable analytics, with a culture that values remote flexibility and strong ownership.