Senior Software Engineer - Husky Storage

Datadog, Inc.; Boston (MA); On-site; USD 192,000 - 240,000; Full time

Benefits offered by this job

New hire stock equity (RSUs)

Employee stock purchase plan (ESPP)

Continuous professional development

Mentor and buddy program

Inclusive culture and mental health

Global benefits

401(k) plan and match

Vacation and wellness reimbursement

Job summary

Datadog is seeking a Senior Software Engineer to join Husky, our petabyte-scale distributed storage engine, to evolve the platform toward Parquet and DataFusion while supporting AI-driven workloads. You will collaborate closely with Husky Ingestion, Metadata, and Platform teams and contribute to open-source efforts.

You will work with teams across the stack to optimize storage, reduce costs, and enable scalable analytics, with a culture that values remote flexibility and strong ownership.

Qualifications

• Experience with large-scale distributed systems and storage engines.
• Strong fundamentals in columnar/file storage formats or comparable systems.
• Bonus: Go, Java, or Rust; compaction or compression schemes; object storage at scale; Kafka and stream processing; Kubernetes.

Responsibilities

• Migrate storage format to Parquet and re-imagine compaction on a DataFusion-based substrate.
• Drive cost-reduction work: block-level index compression, row-level object integration, storage tiering/caching.
• Scale the solution to manage event versioning at scale using temporal-database techniques.
• Operate and influence how to best run the service going forward.
• Collaborate across write path (Ingestion) and handoff to Lakehouse for batch/analytical workloads.

Skills

Distributed systems

Storage engines

Parquet/ORC/Arrow

DataFusion

Go/Java/Rust

S3/GCS/Azure Blob

Kafka

Kubernetes

Tools

Kafka

Kubernetes

S3/GCS/Azure Blob

Job description

Husky is what we call the distributed, petabyte-scale columnar event store at the heart of our Event Platform, which powers dozens of the company’s most popular products – Logs, RUM, APM, Cloud Network Monitoring, Netflow, and many more. Husky was built from the ground up at the company to store and query massive volumes of event data at low cost, with data fully queryable within seconds of arrival.

As a Senior Software Engineer on this team, you'll work hands-on with a distributed storage system at petabyte scale, and help us evolve our platform to support the next level of scale that will power the AI era. We will move from a custom storage format towards open standards (Parquet, Arrow, DataFusion) enabling on prem solutions, increasingly complex analytical/batch workloads, and powering AI observability, sustainably into the future. You'll partner closely with Husky Ingestion, Metadata, and Platform, and increasingly with Lakehouse on the Iceberg-compatible boundary between Husky and analytical/batch workloads. You will also work with the open source community to build the next generation of reusable software solutions that can be operated at scale.

*the company has a culture that is increasingly rare in mature technology companies. We are an engineering driven, bottom up, high agency/high responsibility culture–where everyone is empowered and expected to drive the technological and strategic direction of the business over time. We are a blameless culture where we believe supporting each other through challenges produces the best outcomes over time and our track record corroborates this. We are open source champions. We believe in the flexibility of working remotely but also the value of bringing people together across our various offices worldwide–and our offices are nice places to work.*

What You’ll Do:

• Drive Husky to the next level of adoption by migrating our storage format to open source Parquet and re-imagine compaction on a format-agnostic, DataFusion-based substrate

• Drive concrete cost-reduction work: block-level inverted index compression, combining the database with external objects at the row level, and storage tiering/caching

• Scale our solution to enable customers to manage event versioning at scale, using temporal-database techniques.

• Play an influential role in operating and deciding how to best operate our service going forward.

• Work across the write path (with Ingestion) and toward a cleaner Iceberg-based handoff to ADP/Lakehouse for batch/analytical workloads

Who You Are:

• Enthusiastic and experienced with large-scale distributed systems and/or storage engines

• Comfortable with improving and operating large distributed systems that customers rely on for critical business functions

• Strong fundamentals in columnar/file storage formats (Parquet, ORC, Arrow) or comparable systems

• Bonus: experience with Go, Java, or Rust; with compaction or compression/encoding schemes (RLE, dictionary, delta, gorilla); with object storage (S3/GCS/Azure Blob) at scale, including caching and cost optimization; with Kafka and stream processing; or with Kubernetes

*the company values people from all walks of life. We understand not everyone will meet all the above qualifications on day one. That's okay. If you’re passionate about technology and want to grow your skills, we encourage you to apply.*

Benefits and Growth:

• New hire stock equity (RSUs) and employee stock purchase plan (ESPP)

• Continuous professional development, product training, and career pathing

• Intradepartmental mentor and buddy program for in-house networking

• An inclusive company culture, access to Inclusion Talks

• Free, global mental health benefits for employees and dependents age 6+

• Competitive global benefits

*Benefits and Growth listed above may vary based on the country of your employment and the nature of your employment with the company.*

the company offers a competitive salary and equity package, and may include variable compensation. Actual compensation is based on factors such as the candidate's skills, qualifications, and experience. In addition, the company offers a wide range of best in class, comprehensive and inclusive employee benefits for this role including healthcare, dental, parental planning, and mental health benefits, a 401(k) plan and match, paid time off, fitness reimbursements, and a discounted employee stock purchase plan.

The reasonably estimated yearly salary for this role at the company is:

$192,000—$240,000 USD

About the company:

the company is the leading observability and security platform for the AI era, providing businesses with unified visibility across the technology stack to manage complexity at scale. It brings applications, infrastructure, data, models, and security into one place, using AI to detect and resolve issues before they impact customers. Trusted globally by Fortune 500 companies and high-growth AI leaders, the company enables businesses to move faster with clarity and confidence. Learn more about #DatadogLife on Instagram, LinkedIn, and the company Learning Center.

Equal Opportunity at the company:

the company is proud to offer equal employment opportunity to everyone regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, veteran status, and other characteristics protected by law. We also consider qualified applicants regardless of criminal histories, consistent with legal requirements. Here are our Candidate Legal Notices for your reference.

the company endeavors to make our Careers Page accessible to all users. If you would like to contact us regarding the accessibility of our website or need assistance completing the application process, please complete this form. This form is for accommodation requests only and cannot be used to inquire about the status of applications.

Privacy and AI Guidelines:

Any information you submit to the company as part of your application will be processed in accordance with the company’sApplicant and Candidate Privacy Notice. For information on our AI policy, please visit Interviewing at the company AI Guidelines.