Senior Reliability Engineer (Remote)

Kohl's; Menomonee Falls (WI); On-site; USD 120,000 - 150,000; Full time

Job summary

Kohl’s is seeking a Senior Reliability Engineer to ensure resilience and availability of systems, work with development teams, and design for failure. You will drive SLO adoption, incident response, and automation to improve reliability and efficiency across product teams.

Responsibilities include on-call duties, RCAs, and capacity planning guidance, leveraging tools like CloudWatch and OpenTelemetry. Collaboration with product managers and engineering teams is essential.

Qualifications

• Bachelor’s degree in MIS, CS or related field.
• 4+ years of software development experience.
• Strong programming in Java, Python, Go or Node.js.
• Experience with multi-region troubleshooting and performance tuning.

Responsibilities

• Drive error budget and SLO adoption across products.
• Lead incident response and root-cause analysis to improve reliability.
• Automate processes to improve operational excellence.
• Guide reliability, observability, and efficiency across product teams.
• Identify toil and automate to reduce risk.
• On-call rotation for production incidents and RCAs.

Skills

Programming languages

Cloud platforms

System architecture

Incident response

Chaos engineering

Education

Bachelor's Degree in MIS/CS

Tools

CloudWatch

Grafana

Prometheus

OpenTelemetry

Job description

About The Role

As Senior Reliability Engineer, you will ensure the resilience and availability of Kohl’s systems and applications, collaborate closely with development teams, contribute to architectural designs, conduct risk assessments and design for failure, and implement robust monitoring and failover mechanisms.

What You’ll Do

• Drive error budget and Service Level Objective (SLO) adoption across products
• Drive incident response efforts, perform root cause analysis and implement preventative measures to enhance system reliability
• Establish consistent practices that elevate Kohl’s operational excellence through automation and process improvements
• Follow software lifecycle and drive reliability, observability, and efficiency across product teams within an assigned domain
• Identify repeated toil and find opportunities for automation and risk reduction
• On-call on a rotation to respond to production incidents and conduct blameless retros and root-cause analyses (RCAs) to drive a culture of continuous improvements
• Proactively identifies failures before they cause outages using chaos engineering techniques such as edge cases, failure modes and design review
• Advise on capacity planning and provide continuous assessments on systems behavior and consumption
• Work with product managers to identify and prioritize work for reliability best practices (i.e., leveraging SLIs/SLOs/Error Budgets)
• Mentors and assists engineers on the team
• Additional tasks may be assigned

Required

• Bachelor’s Degree or equivalent in MIS, Computer Science or related field
• 4+ years of experience in software development
• Strong programming skills in one or more languages (Java, Python, Go or Node.js)
• In-depth knowledge of systems architecture, operating system internals and network fundamentals
• In-depth knowledge of application design patterns, event-driven architecture, database schemas, and testing strategies
• Experience with multi-region application troubleshooting and performance tuning
• Working experience on IBM Mainframe (TSO/ISPF, JCL, Batch job scheduling, transaction processing (CICS) and Security (RACF)
• Working experience with one cloud platform (GCP, AWS, or Azure)
• Working experience with monitoring techniques and tools (e.g., CloudWatch, Grafana, Prometheus, OpenTelemetry, Tracing)

Preferred

• In-depth knowledge of containerization and container orchestration (e.g., Docker, Kubernetes, Rancher)
• Experience with one or more configuration management systems (e.g., Chef, Ansible, Puppet)
• Working experience with Elixir
• Passion for and experience with AI and ML methodologies (MLOps)
• Experience writing Infrastructure as code (e.g., Terraform, OpenTofu)