Senior Lead Engineer – Telemetry

BrickRed Systems; Redmond (WA); On-site; USD 120,000 - 160,000; Full time

Job summary

BrickRed Systems is seeking a Senior Lead Engineer – Telemetry to build, analyze, and troubleshoot telemetry across Azure environments. You will work with C#/.NET, KQL, and cloud-native integrations while improving observability and reliability.

The role involves incident investigation, cross-service troubleshooting, and documenting evidence for internal stakeholders. On-site in Redmond, WA with advanced tooling and collaboration.

Qualifications

• Advanced telemetry and observability skills.
• Strong proficiency with KQL and Azure tooling.
• Experience with distributed systems and incident response.

Responsibilities

• Analyze telemetry to identify trends, anomalies, and failures using KQL.
• Work with Geneva, Azure Monitor, Application Insights, and logging/alerting.
• Investigate production incidents and cross-service issues.
• Reconstruct customer journeys and failure timelines from telemetry.
• Develop and maintain C#/.NET apps in C# 14/.NET 10.
• Collaborate with Azure DevOps, Git workflows, YAML pipelines.

Skills

KQL

Azure Monitor

Application Insights

C#/.NET

Azure DevOps

REST APIs

Entra ID/Managed identities

GitHub Copilot

Agent workflows

Job description

We are seeking a highly skilled Senior Lead Engineer – Telemetry to build, analyze, and troubleshoot telemetry and observability solutions across complex, distributed Azure environments. This role combines modern C#/.NET engineering, advanced Kusto Query Language (KQL), production incident investigation, Azure monitoring, and agentic development.

The ideal candidate can investigate complex customer-impacting issues by connecting telemetry across services, reconstructing customer journeys and failure timelines, and producing clear, reproducible technical evidence. You will work closely with engineering, DevOps, security, and product teams to improve system reliability, observability, and operational excellence.

Key Responsibilities

• Analyze complex telemetry using advanced Kusto Query Language (KQL) to identify trends, anomalies, failures, and performance issues.
• Work extensively with Geneva, Azure Monitor, Application Insights, distributed logging, monitoring, and alerting.
• Investigate production incidents and perform cross-service troubleshooting across distributed systems.
• Reconstruct customer journeys and failure timelines using telemetry and customer-related evidence.
• Develop and maintain modern C#/.NET applications, including working in C# 14/.NET 10 codebases.
• Work with Azure DevOps, including work items, Git workflows, YAML pipelines, service connections, and execution artifacts.
• Build and troubleshoot REST APIs and cloud-native integrations using Entra ID, managed identities, workload identity federation, and Azure Key Vault.
• Leverage agentic development tools such as GitHub Copilot, Claude, or equivalent AI coding/engineering tools.
• Design reusable agent workflows using SKILL.md instructions, MCP/tool integrations, validation frameworks, safety boundaries, and human-review checkpoints.
• Partner with engineering and operations teams to improve observability, reliability, alerting, and incident response.
• Produce clear, reproducible technical evidence and documentation with limited supervision.
• Ensure telemetry and customer-related evidence are handled according to privacy, security, compliance, and least-privilege requirements.

Required Qualifications

• Strong experience as a Senior Software Engineer, Telemetry Engineer, Observability Engineer, or similar technical role.
• Advanced proficiency in Kusto Query Language (KQL).
• Hands‑on experience with Geneva, Azure Monitor, Application Insights, distributed logging, monitoring, and alerting.
• Proven experience investigating production incidents and cross-service failures in distributed environments.
• Strong modern C# and .NET development experience, including the ability to work effectively in a C# 14/.NET 10 repository.
• Experience with Azure DevOps, Git, YAML pipelines, service connections, work items, and execution artifacts.
• Strong understanding of REST APIs, Entra ID authentication, managed identity, workload identity federation, and Azure Key Vault.
• Experience using AI/agentic development tools such as GitHub Copilot, Claude, or equivalent.
• Ability to design structured and reusable agent workflows, including tool integration, validation, safety controls, and human-review mechanisms.
• Excellent written communication and ability to independently document technical findings and evidence.
• Strong understanding of privacy, security, data protection, and least-privilege principles when working with telemetry and customer data.

Preferred Qualifications

• Experience with Dynamics 365 Contact Center, Dynamics 365 Customer Service, or Dataverse.
• Experience with Power Platform, Power Automate, custom connectors, DLP, and Configuration Migration Tool.
• Knowledge of performance engineering, scalability, resilience, and chaos-testing concepts.
• Experience with OpenTelemetry, service health monitoring, and SLO/SLA analysis.
• Relevant Azure, DevOps, Security, or Power Platform certifications.

About BrickRed Systems

BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, healthcare, analytics, finance, operations, and business transformation domains.

With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions.

BrickRed Systems fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers.