Member of Technical Staff, Observability

Dipp AI Technologies, Inc.

New York (NY)

Hybrid

USD 180,000 - 230,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Equity
Health, dental and vision coverage
Company hardware
Learning budget
401(k)

Job summary

Dipp AI Technologies is seeking a senior role to design the telemetry architecture for long-running, multi-agent workflows across models, tools, data boundaries and enterprise systems.

The role focuses on preserving causality, distinguishing proposed actions from committed ones, and enabling redaction and access control for sensitive telemetry. You will work across the Immutable Audit Ledger and data governance layers, collaborating with a New York-based founding team.

Qualifications

  • 6+ years building production observability or data-infrastructure products.

Responsibilities

  • Design trace and event schemas connecting directives, identity, policy decisions, model routes, tool calls, costs and outcomes.
  • Build ingestion and correlation systems for asynchronous multi-provider agent runs while preserving causality and tenant boundaries.
  • Create operator views for live execution, failed gates, policy denials, cost anomalies and evidence-package export.
  • Define retention, redaction and access-control behavior for telemetry with sensitive enterprise context.
  • Instrument services with service-level objectives, cardinality controls and failure diagnostics.
  • Turn incidents and customer investigations into product primitives, not dashboards.

Skills

Distributed tracing
Event pipelines
High-cardinality data
Multi-tenant access controls
Incident communication
OpenTelemetry

Education

CS / Engineering degree

Tools

OpenTelemetry
Tracing backends
Streaming platforms
Columnar analytical stores

Job description

Build the evidence layer that lets operators reconstruct every agent decision across models, tools, data boundaries and enterprise systems.

Observability in Orcher is not a dashboard added after execution. It is the operating record that connects a human directive to the identity used, policies evaluated, model and compute route selected, tools called, cost incurred and outcome committed.

You will design the telemetry architecture for long-running, multi-agent workflows that cross frontier APIs, open-weight models, private infrastructure and enterprise systems of record. The hard problems include preserving causality across asynchronous boundaries, distinguishing a proposed action from a committed one, keeping high-cardinality evidence queryable, and making partial failure legible without weakening customer data controls.

This role works across the Immutable Audit Ledger, Cost Governance and Data Control Gateway. Your work will help a security leader investigate an anomalous action, a finance owner explain model spend, and an operator determine whether a workflow completed under the authority and limits originally assigned to it.

What you will do
  • Design trace and event schemas that connect directives, role identity, policy decisions, model routes, tool calls, costs and committed outcomes
  • Build ingestion and correlation systems for asynchronous, multi-provider agent runs without losing causality or tenant boundaries
  • Create operator views for live execution, failed gates, policy denials, cost anomalies and evidence-package export
  • Define retention, redaction and access-control behavior for telemetry that may contain sensitive enterprise context
  • Instrument Orcher services with useful service-level objectives, cardinality controls and failure diagnostics
  • Turn incidents and customer investigations into better product primitives, not one-off dashboards
What we look for
  • Deep experience with distributed tracing, event pipelines, metrics, logs and high-cardinality operational data
  • Strong systems judgment around ordering, retries, idempotency, partial failure and clock or trace discontinuities
  • Ability to model evidence for security, audit and operations audiences without confusing telemetry with proof
  • Experience designing multi-tenant access controls and data-retention boundaries
  • Clear technical writing and calm incident communication
Experience
  • 6+ years building production observability, distributed-systems or data-infrastructure products
  • Shipped systems using technologies such as OpenTelemetry, tracing backends, streaming platforms or columnar analytical stores
  • Experience operating a platform through real incidents and using the findings to improve architecture
Education
  • Degree in computer science, engineering or a related field, or equivalent depth demonstrated through shipped systems
  • Depth in distributed systems, telemetry or data engineering counts in place of a specific degree.
Compensation and benefits
  • Equity with a standard four-year vest and early-stage upside
  • Health, dental and vision coverage
  • Company-provided hardware and a budget for technical research
  • Direct ownership of a foundational Orcher component
  • Hybrid collaboration with the founding team in New York
  • 401(k) with company contribution and an annual learning budget.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff Engineer — Observability & Telemetry
Staff Engineer — Observability & Telemetry

Dipp AI Technologies, Inc. • New York (NY)

Hybrid
USD 180,000 - 230,000
Equity
Health, dental and vision coverage
Company hardware
+2
Forward Deployed Engineer, Orcher
Forward Deployed Engineer, Orcher

Dipp AI Technologies, Inc. • New York (NY)

On-site
USD 120,000 - 180,000
Equity
Health, dental and vision coverage
Travel and field expenses covered
+3
Technical Writer, Trust & Evidence
Technical Writer, Trust & Evidence

Dipp AI Technologies, Inc. • New York (NY)

On-site
USD 70,000 - 100,000
Equity (4-year vest)
Health, dental and vision coverage
Learning budget for professional dev
+2
Field Deployment Engineer - Enterprise Integrations
Field Deployment Engineer - Enterprise Integrations

Dipp AI Technologies, Inc. • New York (NY)

Hybrid
USD 120,000 - 180,000
Equity
Health, dental and vision coverage
Travel and field expenses covered
+3
Member of Technical Staff, Cost Governance
Member of Technical Staff, Cost Governance

Dipp AI Technologies, Inc. • New York (NY)

Hybrid
USD 150,000 - 230,000
Equity vesting
Health coverage
Hardware budget
+3
Staff Engineer, Real-Time AI Cost Governance
Staff Engineer, Real-Time AI Cost Governance

Dipp AI Technologies, Inc. • New York (NY)

Hybrid
USD 150,000 - 230,000
Equity vesting
Health coverage
Hardware budget
+3
VP of Engineering, Control Plane
VP of Engineering, Control Plane

Dipp AI Technologies, Inc. • New York (NY)

Hybrid
USD 190,000 - 320,000
Executive equity grant
Full medical, dental, and vision
401(k) with company contribution
+2
Solutions Architect, Enterprise AI
Solutions Architect, Enterprise AI

Dipp AI Technologies, Inc. • New York (NY)

On-site
USD 150,000 - 230,000
Equity vesting
Medical coverage
Dental coverage
+5
Member of Technical Staff, Evidence Systems
Member of Technical Staff, Evidence Systems

Dipp AI Technologies, Inc. • New York (NY)

On-site
USD 140,000 - 190,000
Equity
Medical, dental, and vision coverage
401(k) with company contribution
+2
Senior Engineering Manager, Observability Engineering
Senior Engineering Manager, Observability Engineering

Temporal • United States

Remote
USD 180,000 - 240,000