Telemetry & AI Evaluation Engineer

Siri InfoSolutions, Inc.; Charlotte (NC); On-site; USD 100,000 - 120,000; Full time

Job summary

Siri InfoSolutions, Inc. in Charlotte, NC is seeking an Observability & Evaluation Engineer for a 12+ month, full-time assignment.

You will implement telemetry, traces, dashboards, alerts, and readiness evidence to support priority agent releases, while performing LLМ and agent evaluations, metrics analysis, and production operations. The role emphasizes Python-based tooling, test automation, and prompt/model performance analysis within a banking-focused context.

Qualifications

• Mid-Senior level with 12+ months project exposure.
• Experience implementing telemetry, traces, and dashboards for releases.

Responsibilities

• Implements telemetry, traces, dashboards, evaluation suites, alerts, service objectives, runbooks, and readiness evidence for priority agent releases.
• LLM and agent evaluation; tracing and telemetry; metrics and dashboards; alerting; SLOs; test automation; prompt and model performance analysis; Python; production operations.

Skills

Telemetry

Tracing

Dashboards

SLOs

Test automation

Prompt analysis

Model performance

Production ops

Runbooks

Tools

Python

Job description

Siri InfoSolutions, Inc. in Charlotte, NC is seeking an Observability & Evaluation Engineer for a 12+ month, full-time assignment.

You will implement telemetry, traces, dashboards, alerts, and readiness evidence to support priority agent releases, while performing LLМ and agent evaluations, metrics analysis, and production operations. The role emphasizes Python-based tooling, test automation, and prompt/model performance analysis within a banking-focused context.