LLM Systems Data Scientist — Evaluation, Robustness & Forecasting

ScienceLogic; United States; On-site; USD 120,000 - 180,000; Full time

Job summary

ScienceLogic is seeking a data scientist to advance our suite of locally-hosted language models and their evaluation framework. You will design evaluation harnesses, measure grounding and reliability, and drive cross‑functional improvements in an enterprise setting with security and compliance constraints.

You will collaborate with data scientists, ML/inference engineers, frontend, and product teams to turn interaction data into actionable insights, forecasting, anomaly detection, and robust

Qualifications

• Experience designing evaluation frameworks for ML systems.
• Strong background in NLP and LLM evaluation metrics.
• Ability to work with cross-functional teams in an enterprise.
• Experience with deployment and monitoring of ML models.

Responsibilities

• Design and own evaluation harnesses for LLM and agent outputs.
• Build LLM-as-judge pipelines; validate judges against human labels and control for bias.
• Define and track response-quality metrics: faithfulness, groundedness, relevance, and instruction-following.
• Curate, version, and grow evaluation datasets as the product surfaces evolve.
• Benchmark the models to decide task allocations and cost implications.
• Red-team the system: prompt injection, jailbreaks, edge-case discovery.
• Design chaos and stress tests for reliability under adverse conditions.
• Characterize failure modes and feed them back into guardrails and regression coverage.
• Evaluate retrieval quality and agent trajectories: recall@k, MRR, context precision/recall.
• Assess intent classification and routing quality as measurable components.
• Build production signals for forecasting, anomaly detection, and early-warning indicators.
• Deploy, monitor, recalibrate models as data shifts, define actionable metrics.

Skills

LLM evaluation

Experimentation design

Data analysis

Python

Education

Master's degree in CS/Statistics or related field

Tools

PyTorch

scikit-learn

Job description

ScienceLogic is seeking a data scientist to advance our suite of locally-hosted language models and their evaluation framework. You will design evaluation harnesses, measure grounding and reliability, and drive cross‑functional improvements in an enterprise setting with security and compliance constraints.

You will collaborate with data scientists, ML/inference engineers, frontend, and product teams to turn interaction data into actionable insights, forecasting, anomaly detection, and robust