AI Product Engineer - ClickStack

ClickHouse; Germany (OH); On-site; USD 120,000 - 160,000; Full time

Benefits offered by this job

Flexible work environment

Healthcare contributions

Equity in the company

$500 Home office setup

Global Gatherings

Generous time off

Job summary

ClickHouse is looking for an AI Product Engineer to enhance their observability platform. This role involves building agents to investigate incidents, ensuring a smooth developer experience, and collaborating with the open-source community.

The ideal candidate will have extensive experience in software engineering, particularly with LLM systems, SQL proficiency, and strong backend skills. Benefits include a flexible work environment, healthcare contributions, and equity options.

Qualifications

• 5+ years of software engineering experience, including 1-2 years on LLM-powered systems.
• Ability to think in production terms, such as latency and cost per task.
• Experience designing skills and working with open-source projects.

Responsibilities

• Build agents that investigate incidents and use ClickStack as their substrate.
• Write a library of reusable skills that help with debugging and incident response.
• Collaborate with OSS contributors to debug and enhance the product.

Skills

Building agents

Context engineering

Tool design

Developer Experience (DX)

SQL proficiency (ClickHouse)

Backend skills (TypeScript/Node.js, Python)

Docker

Kubernetes

Agent observability

Event ingestion pipelines

Job description

We’re hiring an AI Product Engineer to build agentic capabilities on top of a petabyte-scale observability platform, with a focus on developer experience.

Responsibilities

• If you’ve been building agents, designing skills, and wiring up MCP servers — and you’ve hit the limits of generic copilots for production work — we’d like to talk.
• Build agents that investigate incidents. They surface anomalies, answer “why is production broken?”, and use ClickStack as their substrate.
• Write skills, not just prompts. Build a library of reusable skills that captures how our team debugs, finds root causes, writes ClickHouse queries, and runs incident response, so agents pick up the right playbook instead of starting from scratch.
• Own the agent stack end-to-end. Context engineering, tool design, evals, tracing, cost. You’re responsible for whether the agent works in production.
• Make ClickStack a great place to run AI workloads. Build the MCP servers, SDKs, and integrations that let customers’ agents read telemetry, take action, and stay observable themselves.
• Work in the open. Collaborate with OSS contributors and customers, debug their problems alongside them, and feed what you learn back into the product.
• Tackle the hard parts: latency, cost, context window limits, eval coverage, hallucinations on real telemetry.

Benefits

• Flexible work environment – ClickHouse is a globally distributed company and remote‑friendly. We currently operate in 20 countries.
• Healthcare – Employer contributions towards your healthcare.
• Equity in the company – Every new team member who joins our company receives stock options.
• A $500 Home office setup if you’re a remote employee.
• Global Gatherings – We believe in the power of in‑person connection and offer opportunities to engage with colleagues at company‑wide offsites.
• Time off – Flexible time off in the US, generous entitlement in other countries.

Qualifications

• You’ve been building agents long enough to have opinions — about context engineering, tool design, when to use a skill vs. a tool, what evals catch and miss, and where popular frameworks break down.
• You do well with ambiguity and ownership.
• You think in production terms: p99 latency, cost per task, whether the system still works next week without intervention.
• You care about developer tools and have a clear sense of what good DX looks like.
• SQL proficiency – you can write ClickHouse queries directly.
• 5+ years of software engineering experience, including 1–2 years on LLM‑powered systems or agents in production.
• Experience with MCP: building servers, designing tools, and thinking through auth, scoping, and observability for agentic systems.
• Strong backend skills in TypeScript/Node.js and/or Python. Comfortable in both, even if one is primary.
• Comfort with Docker and Kubernetes.
• Hands‑on experience building agents: multi‑step tool use, planning, memory, error recovery. You’ve shipped them and dealt with the failure modes.
• Active in open source and the developer community.
• Strong evals practice: golden sets, LLM‑as‑judge, regression detection.
• Experience designing skills (Markdown‑based workflow encodings, Anthropic‑style or similar) and a clear view on when a skill, a tool, or both is the right fit.
• Built or operated production agents in observability, incident response, or SRE.
• Strong opinions on agent observability – tracing, cost attribution, eval pipelines, OpenTelemetry for agents – and ideas on how to improve it.
• Experience with prompt caching, context compaction, or other techniques relevant to running agents on production telemetry volumes.
• Experience with columnar databases and event ingestion pipelines.
• Contributed to or maintained an open source AI/agent project.
• Familiarity with Go, Rust, or other systems languages for integrations and high‑throughput infra.