Senior Observability Platform Engineer - Alerting

CACI International; Denver (CO); On-site; USD 90,000 - 190,000; Full time

Benefits offered by this job

Healthcare

Wellness programs

Financial planning

Retirement benefits

Family support

Continuing education

Time off benefits

Job summary

CACI International Inc. seeks a Senior Observability Platform Engineer focused on Grafana, alerting, and SRE to transform telemetry into actionable dashboards and reliable incident workflows.

You will define SLIs/SLOs, implement governance, and enable stakeholders across platform teams to observe and act on data. The role requires hands-on Grafana administration, PromQL/LogQL proficiency, and collaboration with IAM, security, and DevOps teams to ensure secure, scalable observability across

Qualifications

• Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field; equivalent experience may substitute.
• 7+ years of experience in SRE, observability engineering, monitoring engineering, platform operations, DevOps, or production support.
• 3+ years of advanced Grafana administration, dashboard development, and alerting experience.
• Strong experience defining and operating SLIs, SLOs, error budgets, alerting standards, incident response, and post-incident improvement processes.
• Demonstrated ability to create usable dashboards and alerts from metrics, logs, traces, and event data.
• Proficiency in PromQL and/or LogQL; experience with Elastic Query DSL/KQL is strongly desired.
• Experience integrating alerts with ServiceNow, GitLab, webhooks, ticketing systems, and enterprise notification workflows.
• Experience with Kubernetes or OpenShift and cloud-native operational troubleshooting.
• Hands-on experience with Git, GitLab CI/CD, Grafana provisioning, GitOps, and configuration-as-code.

Responsibilities

• Grafana Engineering and Visualization: Deploy, administer, secure, and maintain Grafana; build role-based dashboards for multiple teams.
• Alerting and Incident Workflow Engineering: Design, tune, and maintain alerts for availability, performance, errors, security events, and SLO violations.
• SRE, Service Health, and Operational Readiness: Define and maintain SLIs/SLOs; build meta-monitoring dashboards for health metrics.
• Query, Analysis, and Consumer Enablement: Write and optimize PromQL, LogQL, and other queries; enable RED-method dashboards.
• Security, Governance, and Delivery Practices: Implement access controls and ensure compliant data handling and least-privilege access.

Skills

SRE experience

Observability

Monitoring engineering

DevOps

Production support

Education

Bachelor’s degree or equivalent

Tools

Grafana

PromQL

LogQL

Elastic Query DSL/KQL

Kubernetes/OpenShift

Git

GitLab CI/CD

Grafana provisioning

GitOps

Configuration-as-code

Job description

Job Title: Senior Observability Platform Engineer - Alerting

Job Category: Information Technology

Time Type: Full time

Minimum Clearance Required to Start: TS/SCI with Polygraph

Employee Type: Regular

Percentage of Travel Required: Up to 10%

Type of Travel: Continental US

Anticipated Posting End: 3/1/2027

The Opportunity

The Senior Observability Platform Engineer—Grafana, Alerting, and SRE Focus is responsible for turning telemetry into actionable dashboards, alerts, service-level objectives, incident workflows, and reliable operational outcomes. This role combines telemetry, alert engineering, platform reliability engineering, incident-management support, and observability user enablement. The role ensures that platform operators, engineers, security teams, leadership, and FinOps stakeholders can use data effectively through governed dashboards, actionable alerts, and repeatable operational processes.

Responsibilities

• Grafana Engineering and Visualization
  • Deploy, administer, secure, and maintain Grafana as the primary visualization and alerting interface.
  • Build and maintain role-based dashboards for platform engineering, SRE, security operations, application teams, FinOps, and leadership.
• Alerting and Incident Workflow Engineering
  • Design, implement, tune, and maintain alerts for availability, performance, saturation, errors, security events, telemetry gaps, storage issues, and SLO violations.
  • Establish alert standards, severity definitions, deduplication, grouping, inhibition, routing, escalation, silencing, suppression, and maintenance-window practices.
• SRE, Service Health, and Operational Readiness
  • Define, implement, and maintain SLIs, SLOs, error budgets, service-health standards, and reliability reporting for and onboarded services.
  • Build and maintain meta-monitoring dashboards, including pipeline throughput, dropped samples, query latency, ingestion rate, storage utilization, node health, alerting status, and identity-provider health.
• Query, Analysis, and Consumer Enablement
  • Write and optimize PromQL, LogQL, TraceQL, Elastic Query DSL/KQL, or other relevant query languages based on the approved platform architecture.
  • Partner with application teams to build RED-method dashboards, service-level monitoring, trace-to-log correlation, and deployment-aware operational views
• Security, Governance, and Delivery Practices
  • Work with IAM, security, and data-governance stakeholders to implement appropriate dashboard, query, and administrative access controls.
  • Ensure sensitive telemetry is displayed and accessed in accordance with approved data-handling and least-privilege requirements.

Qualifications

Required:

• Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field; equivalent experience may substitute.
• 7+ years of experience in SRE, observability engineering, monitoring engineering, platform operations, DevOps, or production support.
• 3+ years of advanced Grafana administration, dashboard development, and alerting experience.
• Strong experience defining and operating SLIs, SLOs, error budgets, alerting standards, incident response, and post-incident improvement processes.
• Demonstrated ability to create usable dashboards and alerts from metrics, logs, traces, and event data.
• Proficiency in PromQL and/or LogQL; experience with Elastic Query DSL/KQL is strongly desired.
• Experience integrating alerts with ServiceNow, GitLab, webhooks, ticketing systems, and enterprise notification workflows.
• Experience with Kubernetes or OpenShift and cloud-native operational troubleshooting.
• Hands-on experience with Git, GitLab CI/CD, Grafana provisioning, GitOps, and configuration-as-code.

Desired:

• Experience with Grafana Enterprise, including advanced RBAC, reporting, incident-management features, and offline licensing in disconnected environments.
• Experience with Elastic, Prometheus/Mimir, Loki, Tempo, OpenTelemetry, and Grafana Alloy.
• Experience building dashboards for security operations, cloud cost management, OpenCost, or executive operational reporting.
• Experience in air-gapped, classified, DoD, federal, or other highly regulated environments.
• Familiarity with NIST SP 800-53, NIST SP 800-137, ICD 503, RMF/ATO evidence, and FIPS constraints.

What You Can Expect

A culture of integrity.

At CACI, we place character and innovation at the center of everything we do. As a valued team member, you’ll be part of a high-performing group dedicated to our customer’s missions and driven by a higher purpose – to ensure the safety of our nation.

An environment of trust.

CACI values the unique contributions that every employee brings to our company and our customers - every day. You’ll have the autonomy to take the time you need through a unique flexible time off benefit and have access to robust learning resources to make your ambitions a reality.

A focus on continuous growth.

Together, we will advance our nation’s most critical missions, build on our lengthy track record of business success, and find opportunities to break new ground — in your career and in our legacy.

Pay Range

There are a host of factors that can influence final salary including, but not limited to, geographic location, Federal Government contract labor categories and contract wage rates, relevant prior work experience, specific skills and competencies, education, and certifications. Our employees value the flexibility at CACI that allows them to balance quality work and their personal lives. We offer competitive compensation, benefits and learning and development opportunities. Our broad and competitive mix of benefits options is designed to support and protect employees and their families.

Since this position can be worked in more than one location, the range shown is the national average for the position.

The proposed salary range for this position is:

$90,300-$189,600

At CACI, you will receive comprehensive benefits such as:

• healthcare
• wellness
• financial
• retirement
• family support
• continuing education
• time off benefits

CACI is anEqualOpportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, age, national origin, disability, status as a protected veteran, or any otherprotectedcharacteristic.