AI Quality & Safety Evaluation Engineer

Capital Rx; Denver (CO); On-site; USD 120,000 - 180,000; Full time

Job summary

Judi Health is seeking an AI Evaluation Engineer to build testing frameworks, metrics, and tooling to assess the safety, reliability, and accuracy of AI models in production. You will translate ambiguous product goals into measurable quality targets and drive standardized evaluation across multiple agent types.

You will own data engineering tasks, platform observability, and evaluation tooling, collaborating with data science and engineering teams to ensure high-quality, reproducible results and

Qualifications

• 4+ years of experience in data engineering, ML engineering, or software engineering.
• Bachelor's or Master's degree in Computer Science, Machine Learning, or a related quantitative field.
• Strong proficiency in Python.
• Strong SQL skills.
• Experience building and maintaining production data pipelines.
• Experience working with at least one cloud platform (AWS preferred).

Responsibilities

• Data Engineering: Build and maintain ETL pipelines for heterogeneous data sources (traces, logs, transcripts, user feedback).
• Platform & Observability: Develop dashboards and monitoring tools for AI quality metrics and integrate evaluations into CI/CD pipelines.
• AI / ML Evaluation Tooling: Apply and extend evaluation patterns and design metrics for stochastic systems using LangSmith and related tools.
• Collaboration: Partner with data science, engineering, and product teams to translate goals into engineering requirements and ensure high usability of tooling.

Skills

Python

SQL

Cloud platforms (AWS)

Data pipelines

Education

Bachelor's or Master's in CS/ML/related field

Tools

LangSmith

Playwright

Job description

Judi Health is seeking an AI Evaluation Engineer to build testing frameworks, metrics, and tooling to assess the safety, reliability, and accuracy of AI models in production. You will translate ambiguous product goals into measurable quality targets and drive standardized evaluation across multiple agent types.

You will own data engineering tasks, platform observability, and evaluation tooling, collaborating with data science and engineering teams to ensure high-quality, reproducible results and