Observability Engineer - Remote

Harbor Freight Tools; Calabasas (CA); On-site; USD 88,000 - 132,000; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

Life insurance

Disability insurance

401k plan

Paid time off

Paid sick time

Job summary

Harbor Freight Tools is seeking an Observability Engineer to design, implement, and administer tools and pipelines for full-stack visibility across infrastructure, apps, and services. You will own the NOC/TOC operations and drive high-signal alerting for rapid incident response.

You will work with Networking, Cloud, and DevOps teams to build, tune, and scale observability practices, dashboards, and KPIs (SLIs/SLOs). This role requires after-hours work and on-call rotation.

Qualifications

• Experience designing and implementing observability platforms across metrics, logs, traces.
• Own and operate the NOC/TOC function end-to-end, including alerts, triage, escalation, and post-incident review.
• Collaborate with Infrastructure, DevOps/SRE, and Application teams.
• Lead capacity planning and performance trending using historical data.

Responsibilities

• Design, deploy, and maintain observability platforms across metrics, logs, traces, and monitors.
• Own and operate the NOC/TOC end-to-end: alerts, triage, escalation, post-incident reviews.
• Develop dashboards and visualizations for technical and business stakeholders.
• Define and report on SLIs, SLOs, error budgets.
• Mentor staff on observability concepts and instrumentation practices.
• Integrate observability data with automation workflows for auto-remediation.
• After-hours work and on-call rotation may be required.

Job description

The Observability Engineer is responsible for designing, building, implementing, and administering the tools and pipelines that give the organization full-stack visibility into its infrastructure, applications, and business services. This role owns the day-to-day operation of the observability platform(s) and the NOC/TOC (Network/Technical Operations Center) function to ensure stability, reliability, availability, and performance of the services the business depends on.

Where traditional monitoring answered, “is it up or down,” this role is expected to build observability that answers “why is it behaving this way” — correlating metrics, logs, traces, and events into actionable insight, and reducing time-to-detect and time-to-resolve across the environment.

Duties and Responsibilities

• Provides expertise and develops observability solutions for:
• Network technology, connections, and cloud networking (SD-WAN, VPC/VNet constructs)
• Directory services (Active Directory, LDAP, Entra ID/Azure AD)
• Messaging and collaboration platforms (Exchange Online, Microsoft 365, Teams)
• eCommerce and customer-facing digital environments
• Remote access and VPN infrastructure
• Wireless, mobile, and endpoint device fleets
• Operating system environments (Windows, Linux)
• Data backup, storage, and disaster recovery systems
• VOIP and unified communications systems
• Virtualized and containerized environments (VMware)
• Public and hybrid cloud infrastructure (AWS, Azure, GCP)
• Critical business services and transaction flows
• Design, deploy, and maintain observability platforms across metrics, logs, traces, synthetic monitoring, and real user monitoring
• Own and operate the NOC/TOC function end-to-end: alert intake, triage, escalation, incident coordination, and post-incident review
• Develop and maintain monitoring/observability standards, naming conventions, and instrumentation guidelines across infrastructure and application teams
• Build and tune alerting logic to reduce noise and alert fatigue; drive toward actionable, high-signal alerting
• Define, track, and report on SLIs, SLOs, error budgets, and other key observability/operational KPIs
• Develop dashboards and visualizations that give both technical teams and business stakeholders a clear picture of service health
• Mentor technical staff and cross-functional teams on observability concepts, instrumentation practices, and tooling
• Develop, document, and continuously improve monitoring and incident response processes and runbooks
• Troubleshoot and resolve issues within the monitoring/observability stack itself (data gaps, false positives, agent/collector health)
• Evaluate, pilot, and recommend new observability tools and platforms
• Partner with Help Desk, Infrastructure, DevOps/SRE, and Application teams to implement monitoring, alerting, and response processes
• Manage the call notification, paging, and escalation system for alerts
• Lead capacity planning, performance trending, and fault prediction using historical observability data
• Integrate observability data with automation/AIOps workflows to enable auto-remediation where appropriate
• Support security operations by feeding relevant telemetry into SIEM/SOC tooling and collaborating on anomaly detection
• After-hours work, shift work, and on-call rotation required

Scope

• Staff supervision and development: No
• Decision making:
• Review data and provide recommendations on alert actions, monitoring and responses.
• Travel: Up to 5%
• Flex Designation: Anywhere

The anticipated salary range for this position is $87,600– $131,500 depending on location, knowledge, skills, education and experience. This position is also eligible for an annual discretionary bonus.

• In addition, we offer comprehensive and competitive benefits to Associates (and their families) such as medical, dental, vision, life insurance, short-term and long-term disability.
• Eligible Associates are able to enroll in our company’s 401k plan.
• Associates will accrue paid time off up to 236 hours per year (inclusive of PTO, floating holidays, and paid holidays).
• Paid sick time up to 80 hours per year unless otherwise required by law.