Observability Engineer

Sepal; Remote; USD 80,000 - 100,000; Full time

Job summary

A leading AI technology company in Philadelphia seeks an Observability Engineer to design complex queries over large telemetry datasets. The ideal candidate will have over 3 years of relevant experience, strong SQL skills, and expertise in tools like Clickhouse and Grafana. Responsibilities include creating synthetic datasets and optimizing query performance to enhance system reliability. This role offers flexible hours and a competitive pay rate of $50 - $85/hr based on experience for a 5–6 week project.

Qualifications

• 3+ years of experience in observability engineer, production engineer, or platform engineer roles.
• Deep expertise in Clickhouse and Grafana, with a focus on large-scale query optimization, schema design, and performance tuning.
• Strong SQL skills: ability to reason through performance problems and spot inefficient query patterns.

Responsibilities

• Design complex, distributed queries over massive log and telemetry datasets.
• Explore creative ways to challenge AI's reasoning ability and log analysis skills.
• Create and manage synthetic datasets that simulate real-world DevOps or cloud infrastructure logs.

Skills

SQL

Clickhouse

Grafana

Query optimization

Distributed systems

Tools

Clickhouse

Grafana

Job description

Overview

The Observability Engineer / Production Engineer will design and optimize complex queries over large telemetry datasets to support AI‑driven incident investigation and system reliability. This role involves creating synthetic datasets simulating cloud infrastructure logs and working extensively with tools like Clickhouse and Grafana. Candidates should have 3+ years of experience in observability or production engineering with strong SQL and query optimization skills.

About Sepal AI

Sepal AI builds the world’s hardest tests for AI grounded in real‑world software systems. We are looking for an Observability Engineer with 3+ years of experience to help us understand, debug, and operate complex production systems at scale. You will work deeply with production logs, metrics, and traces, building tasks and datasets that can teach AI to investigate incidents, analyze performance, and improve system reliability.

What You’ll Do

• Design complex, distributed queries over massive log and telemetry datasets.
• Explore creative ways to challenge AI's reasoning ability and log analysis skills.
• Create and manage synthetic datasets that simulate real‑world DevOps, observability, or cloud infrastructure logs.

Who You Are

• 3+ years of experience in observability engineer, production engineer, or platform engineer roles.
• Deep expertise in Clickhouse and Grafana, with a focus on large‑scale query optimization, schema design, and performance tuning.
• Strong SQL skills: you know how to reason through performance problems and spot inefficient query patterns.

Compensation & Logistics

Pay: $50 – 85/hr depending on experience. Remote, flexible hours. Project timeline: 5–6 weeks.

Keywords

Observability Engineer, Production Engineer, Clickhouse, Grafana, SQL optimization, Distributed systems, Telemetry data, Cloud infrastructure logs, AI debugging, System reliability.