AI Product Engineer: Incident-Response Agents (Remote)

ClickHouse; San Francisco (CA); Remote; USD 112,000 - 168,000; Full time

Benefits offered by this job

Flexible work

Healthcare

Equity

Time off

Home office stipend

Global gatherings

Job summary

ClickHouse, through ClickStack, is hiring to build the AI layer for Observability. You will develop agents that investigate incidents, surface anomalies, and propose root causes while keeping telemetry accessible for on-call engineers.

You will design and ship a library of reusable skills, own the agent stack end-to-end, and collaborate with OSS contributors. This role emphasizes production readiness, cost awareness, and strong DX in a remote-friendly environment.

Qualifications

• 5+ years of software engineering experience, incl 1–2 years on LLM-powered systems in production.
• Strong backend skills in TypeScript/Node.js and/or Python, comfortable in both.
• Hands-on experience building agents with multi-step tool use, planning, memory.
• Experience designing skills and a clear view on when a skill, a tool, or both is the right fit.
• Experience with MCP: building servers, designing tools, and observability for agent systems.
• Strong evals practice: golden sets, regression detection.
• SQL proficiency - write ClickHouse queries directly.
• Comfort with Docker and Kubernetes.
• Active in open source and the developer community.

Responsibilities

• Build agents that investigate incidents, surface anomalies, and propose root causes.
• Write skills and build a library of reusable plays for agents.
• Own the agent stack end-to-end: context engineering, tool design, evals, tracing and cost.
• Make ClickStack capable of running AI workloads with MCP servers and integrations.
• Work in the open, collaborating with OSS contributors and customers.
• Tackle latency, cost, and context window challenges in production.

Skills

LLM-powered systems

TypeScript/Node.js

Python

Agent development

Skills design

MCP

Eval pipelines

SQL (ClickHouse)

Docker

Kubernetes

Open source

Tools

Docker

Kubernetes

ClickHouse

Job description

ClickHouse, through ClickStack, is hiring to build the AI layer for Observability. You will develop agents that investigate incidents, surface anomalies, and propose root causes while keeping telemetry accessible for on-call engineers.

You will design and ship a library of reusable skills, own the agent stack end-to-end, and collaborate with OSS contributors. This role emphasizes production readiness, cost awareness, and strong DX in a remote-friendly environment.