LLMOps Data Analyst - Observability & Quality

Rightsline Inc; El Segundo (CA); On-site; USD 110,000 - 120,000; Full time

Job summary

Rightsline Inc. is seeking a Senior Data Analyst – LLMOps to own observability, evaluation, and quality measurement of AI features across Rightsline’s SaaS platform.

You will instrument the LLM stack end to end, including tracing, prompts, tool calls, latency, token spend, and output quality, building evaluation suites to guide product decisions. You will collaborate across engineering, product, and data teams to ship measurable improvements, with dashboards and reports that drive ship/no-ship

Qualifications

• 5+ years in data analytics, analytics engineering, or data science, including hands-on work with LLM-based systems in production.
• Strong SQL and Python (including pandas and notebook-based analysis), with the discipline to build repeatable pipelines rather than one-off queries.
• Hands-on experience with LLM observability and evaluation tooling such as LangSmith, Langfuse, LangChain/LangGraph, or equivalent.
• Practical understanding of LLM application architecture — prompting, retrieval-augmented generation, tool calling, and agent workflows — and how each of them fails.
• Experience designing evaluation methodology: golden datasets, scoring rubrics, LLM-as-judge grading, human annotation, and A/B testing on production traffic
• Working knowledge of experiment design and statistics, enough to say confidently whether a difference is real
• Ability to work independently, meet deadlines, and be accountable for your work, while operating within established standards
• Excellent written and verbal communication skills, and a strong sense of ownership, urgency, and initiative
• BS in Computer Science, Statistics, Data Science, or equivalent knowledge

Responsibilities

• Instrument Rightsline’s LLM and agent workflows end to end with tracing and structured logging.
• Define and maintain the metrics that describe AI system health — latency, token and cost per request, error and fallback rates, tool-call success, and retrieval hit rates.
• Build and maintain dashboards and alerting so quality and cost regressions surface quickly.
• Design and maintain offline evaluation suites — golden datasets, scoring rubrics, and LLM-as-judge grading.
• Run online evaluations and A/B experiments on production traffic and report rigorous results.
• Track quality across model, prompt, and retrieval changes and maintain benchmark history.
• Partner with engineers on human review and annotation workflows.

Skills

SQL

Python

LLM observability

Data analytics

Communication

Education

BS in Computer Science, Statistics, Data Science, or equivalent

Tools

LangSmith

Langfuse

LangChain/LangGraph

AWS

dbt

Airflow

Git

Looker

Power BI

Tableau

MS SQL

Redis

Job description

Rightsline Inc. is seeking a Senior Data Analyst – LLMOps to own observability, evaluation, and quality measurement of AI features across Rightsline’s SaaS platform.

You will instrument the LLM stack end to end, including tracing, prompts, tool calls, latency, token spend, and output quality, building evaluation suites to guide product decisions. You will collaborate across engineering, product, and data teams to ship measurable improvements, with dashboards and reports that drive ship/no-ship