Senior Observability & Evaluation Engineer - LLM Telemetry

Symphony Solutions; Charlotte (NC); On-site; USD 120,000 - 180,000; Full time

Job summary

Symphony Solutions in Charlotte, NC is seeking an Observability and Evaluation Engineer to join onsite. You will own telemetry pipelines, traces, dashboards, and evaluation suites for priority agent releases, collaborating with cross-functional teams to ensure reliability and performance.

Ideal candidates bring deep Python experience and a track record in production operations, with expertise in LLM evaluation, prompt and model performance analysis, and building runnable runbooks and SLOs.

Qualifications

• Role requires 12+ years of relevant experience.
• Experience delivering telemetry, traces and dashboards.
• Experience with LLM evaluation and agent assessment.
• Strong Python skills for production systems.
• Familiarity with runbooks and readiness evidence.

Responsibilities

• LLM and agent evaluation for priority releases.
• Implement tracing and telemetry across services.
• Develop metrics and dashboards for monitoring.
• Define SLOs and instrumentation for systems.
• Automate tests and validation pipelines.
• Analyze prompt and model performance.
• Support production operations and incident readiness.

Skills

Python

Telemetry

Tracing

Dashboards

LLM evaluation

Test automation

Model performance analysis

Production operations

Prompt engineering

Job description

Symphony Solutions in Charlotte, NC is seeking an Observability and Evaluation Engineer to join onsite. You will own telemetry pipelines, traces, dashboards, and evaluation suites for priority agent releases, collaborating with cross-functional teams to ensure reliability and performance.

Ideal candidates bring deep Python experience and a track record in production operations, with expertise in LLM evaluation, prompt and model performance analysis, and building runnable runbooks and SLOs.