Senior Chaos Engineer: Resilience & Automation Architect

Datadog, Inc.; Paris; Hybrid; EUR 90,000 - 150,000; Full time

Job summary

Datadog, a leader in observability, seeks a Senior Software Engineer to advance chaos engineering and zonal resilience. You will build automation to evacuate and recover workloads safely, design fault-injection capabilities, and craft safeguards for reversible production experiments.

Collaborate across teams to turn resilience findings into verified remediation. You will work on distributed systems, gRPC services, and Kubernetes-based platforms, mentoring others and advancing AI-assisted

Qualifications

• Strong distributed systems fundamentals and the ability to reason about consistency, failure modes, backpressure, idempotency, quorum, retries and recovery.
• Understanding Kubernetes workload lifecycles, including pods, controllers, scheduling, draining and eviction in resilient design.
• Experience designing, building, or operating production systems where safety, availability, and controlled failure handling are important.
• Clear communication of complex technical decisions through design docs, runbooks, postmortems, and cross-functional discussions.
• Comfortable collaborating across teams to identify failure modes and drive resilience improvements.

Responsibilities

• Build zonal-resilience automation that coordinates safe evacuations, switchovers, and recovery with affected services teams.
• Design and build fault-injection systems for production environments, including testing and incident replay.
• Develop safeguards like blast-radius controls, kill switches, validation mechanisms, and rollback paths.
• Build agents and automation to propose failure scenarios, triage results, and connect findings to remediation.
• Lead gamedays from design through execution, findings, remediation tracking, and verification of fixes.
• Design and implement reliable distributed systems, including gRPC services, Kubernetes controllers, and shared components, while mentoring engineers.

Skills

Distributed systems

Kubernetes

gRPC

Chaos engineering

Tools

Kubernetes controllers

gRPC services

Job description

Datadog, a leader in observability, seeks a Senior Software Engineer to advance chaos engineering and zonal resilience. You will build automation to evacuate and recover workloads safely, design fault-injection capabilities, and craft safeguards for reversible production experiments.

Collaborate across teams to turn resilience findings into verified remediation. You will work on distributed systems, gRPC services, and Kubernetes-based platforms, mentoring others and advancing AI-assisted