Telemetry & Observability Engineer - AI Systems & Missions

Matter Intelligence; El Segundo (CA); On-site; USD 170,000 - 210,000; Full time

Benefits offered by this job

100% employer-paid health, dental, and

vision coverage

Job summary

Matter Intelligence is seeking a Telemetry and Observability Engineer to connect flight and mission reality with model behavior through shared telemetry, traces, and reliability mechanisms, reporting to Ignacio Cases Martin. This role focuses on observability across datasets, training runs, models, and environments within a space- and defense-grade AI system.

You will instrument training, inference, and agent workflows, linking mission state with hardware context, time, geometry, and geolocation

Qualifications

• Experience with ML observability, telemetry, time-series systems, distributed training or inference, cloud infrastructure, or production AI platforms.
• Strong hands-on engineering skills in observability, automation, APIs, infrastructure, incident tooling, and systems debugging.
• Understanding of metrics, logs, traces, events, model and data monitoring, service objectives, capacity, release safety, and incident diagnosis.
• Ability to reason about AI failure modes including bad datasets, training instability, stale checkpoints, drift, version mismatch, environment bugs, and agent-tool failure.
• Experience designing telemetry schemas and correlation workflows across multiple systems, time domains, and ownership boundaries.

Responsibilities

• Build a common telemetry model linking mission state, datasets, code, checkpoints, accelerators, model and prompt versions, environment episodes, agent steps, tools, decisions, cost, latency, and feedback.
• Instrument distributed training, data loading, GPU utilization, checkpointing, experiment health, batch and online inference, and model-serving behavior.
• Instrument agent workflows across prompts, context, retrieval, memory, planning, tool calls, graph state, human intervention, evidence, outcomes, and safety controls.
• Connect planned-versus-actual mission state, hardware context, time, geometry, and geolocation to relevant datasets and model results.
• Establish actionable service objectives, alerts, incident interfaces, rollout and rollback signals, and escalation paths across platform, model, environment, and agent failures.
• Build capacity and cost visibility across storage, processing, accelerators, training, evaluation, inference, retrieval, and agent execution.

Skills

ML observability

telemetry

time-series systems

distributed training

cloud infrastructure

production AI platforms

Job description

Matter Intelligence is seeking a Telemetry and Observability Engineer to connect flight and mission reality with model behavior through shared telemetry, traces, and reliability mechanisms, reporting to Ignacio Cases Martin. This role focuses on observability across datasets, training runs, models, and environments within a space- and defense-grade AI system.

You will instrument training, inference, and agent workflows, linking mission state with hardware context, time, geometry, and geolocation