Observability Engineer

Sepal; Remote; USD 80,000 - 100,000; Full time

Job summary

A tech company specializing in AI testing is looking for an Observability Engineer to design complex queries and optimize telemetry datasets. This role requires 3+ years of experience in production or observability engineering, with strong skills in SQL and tools like Clickhouse and Grafana. The flexibility of remote work and competitive pay ranging from $50 to $85 per hour makes it an attractive position for candidates passionate about system reliability and AI debugging.

Qualifications

• 3+ years of experience in observability or production engineering roles.
• Strong SQL skills to diagnose performance issues and optimize queries.
• Experience with creating and managing synthetic datasets.

Responsibilities

• Design complex, distributed queries over massive log and telemetry datasets.
• Explore creative ways to challenge AI's reasoning ability in log analysis.
• Create and manage synthetic datasets to simulate real-world logs.

Skills

SQL optimization

Clickhouse

Grafana

Observability practices

Distributed systems

Tools

Clickhouse

Grafana

Job description

Overview

The Observability Engineer / Production Engineer will design and optimize complex queries over large telemetry datasets to support AI‑driven incident investigation and system reliability. This role involves creating synthetic datasets simulating cloud infrastructure logs and working extensively with tools like Clickhouse and Grafana. Candidates should have 3+ years of experience in observability or production engineering with strong SQL and query optimization skills.

About Sepal AI

Sepal AI builds the world’s hardest tests for AI grounded in real‑world software systems. We are looking for an Observability Engineer with 3+ years of experience to help us understand, debug, and operate complex production systems at scale. You will work deeply with production logs, metrics, and traces, building tasks and datasets that can teach AI to investigate incidents, analyze performance, and improve system reliability.

What You’ll Do

• Design complex, distributed queries over massive log and telemetry datasets.
• Explore creative ways to challenge AI's reasoning ability and log analysis skills.
• Create and manage synthetic datasets that simulate real‑world DevOps, observability, or cloud infrastructure logs.

Who You Are

• 3+ years of experience in observability engineer, production engineer, or platform engineer roles.
• Deep expertise in Clickhouse and Grafana, with a focus on large‑scale query optimization, schema design, and performance tuning.
• Strong SQL skills: you know how to reason through performance problems and spot inefficient query patterns.

Compensation & Logistics

Pay: $50 – 85/hr depending on experience. Remote, flexible hours. Project timeline: 5–6 weeks.

Keywords

Observability Engineer, Production Engineer, Clickhouse, Grafana, SQL optimization, Distributed systems, Telemetry data, Cloud infrastructure logs, AI debugging, System reliability.