Member of Technical Staff, Observability

Dipp AI Technologies; Northern, New York (KY, NY); Hybrid; USD 180,000 - 230,000; Full time

Benefits offered by this job

Equity

Health, dental and vision coverage

Company hardware

Learning budget

401(k)

Job summary

Dipp AI Technologies is seeking a senior role to design the telemetry architecture for long-running, multi-agent workflows across models, tools, data boundaries and enterprise systems.

The role focuses on preserving causality, distinguishing proposed actions from committed ones, and enabling redaction and access control for sensitive telemetry. You will work across the Immutable Audit Ledger and data governance layers, collaborating with a New York-based founding team.

Qualifications

• 6+ years building production observability or data-infrastructure products.

Responsibilities

• Design trace and event schemas connecting directives, identity, policy decisions, model routes, tool calls, costs and outcomes.
• Build ingestion and correlation systems for asynchronous multi-provider agent runs while preserving causality and tenant boundaries.
• Create operator views for live execution, failed gates, policy denials, cost anomalies and evidence-package export.
• Define retention, redaction and access-control behavior for telemetry with sensitive enterprise context.
• Instrument services with service-level objectives, cardinality controls and failure diagnostics.
• Turn incidents and customer investigations into product primitives, not dashboards.

Skills

Distributed tracing

Event pipelines

High-cardinality data

Multi-tenant access controls

Incident communication

OpenTelemetry

Education

CS / Engineering degree

Tools

OpenTelemetry

Tracing backends

Streaming platforms

Columnar analytical stores

Job description

Build the evidence layer that lets operators reconstruct every agent decision across models, tools, data boundaries and enterprise systems.

Observability in Orcher is not a dashboard added after execution. It is the operating record that connects a human directive to the identity used, policies evaluated, model and compute route selected, tools called, cost incurred and outcome committed.

You will design the telemetry architecture for long-running, multi-agent workflows that cross frontier APIs, open-weight models, private infrastructure and enterprise systems of record. The hard problems include preserving causality across asynchronous boundaries, distinguishing a proposed action from a committed one, keeping high-cardinality evidence queryable, and making partial failure legible without weakening customer data controls.

This role works across the Immutable Audit Ledger, Cost Governance and Data Control Gateway. Your work will help a security leader investigate an anomalous action, a finance owner explain model spend, and an operator determine whether a workflow completed under the authority and limits originally assigned to it.

What you will do

• Design trace and event schemas that connect directives, role identity, policy decisions, model routes, tool calls, costs and committed outcomes
• Build ingestion and correlation systems for asynchronous, multi-provider agent runs without losing causality or tenant boundaries
• Create operator views for live execution, failed gates, policy denials, cost anomalies and evidence-package export
• Define retention, redaction and access-control behavior for telemetry that may contain sensitive enterprise context
• Instrument Orcher services with useful service-level objectives, cardinality controls and failure diagnostics
• Turn incidents and customer investigations into better product primitives, not one-off dashboards

What we look for

• Deep experience with distributed tracing, event pipelines, metrics, logs and high-cardinality operational data
• Strong systems judgment around ordering, retries, idempotency, partial failure and clock or trace discontinuities
• Ability to model evidence for security, audit and operations audiences without confusing telemetry with proof
• Experience designing multi-tenant access controls and data-retention boundaries
• Clear technical writing and calm incident communication

Experience

• 6+ years building production observability, distributed-systems or data-infrastructure products
• Shipped systems using technologies such as OpenTelemetry, tracing backends, streaming platforms or columnar analytical stores
• Experience operating a platform through real incidents and using the findings to improve architecture

Education

• Degree in computer science, engineering or a related field, or equivalent depth demonstrated through shipped systems
• Depth in distributed systems, telemetry or data engineering counts in place of a specific degree.

Compensation and benefits

• Equity with a standard four-year vest and early-stage upside
• Health, dental and vision coverage
• Company-provided hardware and a budget for technical research
• Direct ownership of a foundational Orcher component
• Hybrid collaboration with the founding team in New York
• 401(k) with company contribution and an annual learning budget.