LLMOps Data Analyst: Observability & Evaluation

Rightsline; United States; Presencial; USD 120.000 - 160.000; Jornada completa

Descripción de la vacante

Rightsline is seeking a Senior Data Analyst - LLMOps to own observability, evaluation, and quality measurement of our LLM-powered features. You will instrument the LLM stack end to end—traces, prompts, tool calls, latency, token spend—and build evaluation suites that guide engineering and product decisions.

This senior role spans metrics, dashboards, online/offline evaluation, and collaboration across the data, engineering and product teams in a North American context.

Formación

• 5+ years in data analytics, analytics engineering, or data science with hands-on experience in production LLM-based systems.
• Strong SQL and Python skills with ability to build repeatable data pipelines.
• Experience with LLM observability and evaluation tooling and methodologies.

Responsabilidades

• Own observability, telemetry, and quality measurement of Rightsline's LLM-powered features.
• Instrument LLM stack end to end with tracing, prompts, tool calls, latency, and token spend.
• Build evaluation suites and dashboards to translate telemetry into product decisions.
• Run offline and online evaluations, support ship/no-ship decisions, and track quality trends.
• Collaborate with engineering, product and data teams to improve AI features and cost efficiency.

Descripción del empleo

Rightsline is seeking a Senior Data Analyst - LLMOps to own observability, evaluation, and quality measurement of our LLM-powered features. You will instrument the LLM stack end to end—traces, prompts, tool calls, latency, token spend—and build evaluation suites that guide engineering and product decisions.

This senior role spans metrics, dashboards, online/offline evaluation, and collaboration across the data, engineering and product teams in a North American context.