LLMOps Data Analyst: Observability & Evaluation Leader

ADP, Inc.; El Segundo (CA); Sur place; USD 110 000 - 120 000; Plein temps

Résumé du poste

Rightsline seeks a Senior Data Analyst – LLMOps to own observability, evaluation, and quality measurement of AI features across a SaaS platform. You will instrument the LLM stack with tracing, build evaluation suites, and generate dashboards that drive product decisions.

You will push for repeatable data pipelines and robust experiments, partnering with engineering and product teams to improve AI quality, cost efficiency, and reliability across the stack.

Qualifications

• 5+ years in data analytics, analytics engineering, or data science, including hands-on work with LLM-based systems in production.
• Strong SQL and Python (including pandas and notebook-based analysis), with the discipline to build repeatable pipelines rather than one-off queries.
• Hands-on experience with LLM observability and evaluation tooling such as LangSmith, Langfuse, LangChain/LangGraph, or equivalent.
• Practical understanding of LLM application architecture — prompting, retrieval-augmented generation, tool calling, and agent workflows — and how each of them fails.

Responsabilités

• Instrument LLM and agent workflows end to end with tracing and logging; define metrics like latency, token spend, error rates, and tool-call success.
• Design and maintain offline evaluation suites and run online experiments; report results with rigor for ship decisions.
• Build pipelines moving trace, evaluation, and usage telemetry into the analytics stack using SQL and Python.
• Monitor token spend and model usage; identify where changes reduce cost without sacrificing quality.
• Attend and contribute to team meetings and share telemetry/evaluation findings for non-specialists.

Connaissances

SQL

Python

LLM observability

Data analysis

Dashboarding

A/B testing

Formation

BS in Computer Science, Statistics, Data Science, or equivalent

Outils

LangSmith

Langfuse

LangChain/LangGraph

Description du poste

Rightsline seeks a Senior Data Analyst – LLMOps to own observability, evaluation, and quality measurement of AI features across a SaaS platform. You will instrument the LLM stack with tracing, build evaluation suites, and generate dashboards that drive product decisions.

You will push for repeatable data pipelines and robust experiments, partnering with engineering and product teams to improve AI quality, cost efficiency, and reliability across the stack.