Data Scientist, LLM Evaluation & AIOps

Sciencelogic; United States; On-site; USD 120,000 - 155,000; Full time

Benefits offered by this job

401(k) plan with employer match

Flexible Paid Time Off (FTO)

Volunteer Time Off (VTO) - two days a 

Paid parental leave

HQ Office Reston Town Center

Job summary

ScienceLogic is seeking a Data Scientist to join our growing Data Science team. We run a suite of small, locally-hosted language models in production—focusing on evaluation, grounding, and orchestration for reliable outcomes in an enterprise setting.

You will design evaluation harnesses, build prediction and trend capabilities from operational telemetry, and work with ML engineers, frontend, and product while navigating security and compliance constraints.

Qualifications

• Bachelor's or Master's in Data Science, CS, Statistics, Math, or related field or equivalent experience.
• 3+ years in data science, ML, or applied quantitative analysis.
• Strong applied statistics with ability to design experiments and significance tests on noisy outputs.
• Experience building, deploying, and monitoring predictive or time-series models in production.
• Demonstrated work evaluating, analyzing, or improving LLM/NLP systems: eval design and quality measurement.
• Proficiency in Python and SQL; comfortable querying large analytical datasets.

Responsibilities

• Design and own evaluation harnesses for LLM and agentic outputs with rubrics and golden sets.
• Build LLM-as-judge pipelines and calibrate against human labels to control bias and variance.
• Define and track response-quality metrics: faithfulness, grounding, hallucination rate, relevance, and adherence.
• Curate and grow evaluation datasets as surfaces evolve; benchmark models to guide task allocation.
• Develop production prediction and trend capability from operational telemetry.
• Collaborate with ML/inference engineers, frontend, and product in a secure enterprise environment.

Skills

Python

SQL

Statistics

Machine Learning

NLP evaluation

Experiment design

Education

Bachelor's or Master's in a related field

Tools

Git

Jupyter

Docker

Job description

ScienceLogic is seeking a Data Scientist to join our growing Data Science team. We run a suite of small, locally-hosted language models in production—focusing on evaluation, grounding, and orchestration for reliable outcomes in an enterprise setting.

You will design evaluation harnesses, build prediction and trend capabilities from operational telemetry, and work with ML engineers, frontend, and product while navigating security and compliance constraints.