AI Quality & Safety Evaluation Engineer

Capital Rx; New York (NY); On-site; USD 120,000 - 160,000; Full time

Job summary

Judi Health is seeking an AI Evaluation Engineer to build testing frameworks, metrics, and tooling to assess safety, reliability, and accuracy of AI models in production. You will translate product goals into measurable quality targets and collaborate with data science, engineering, and product teams.

Responsibilities include creating evaluation pipelines, dashboards, and tooling, and contributing to a standardized evaluation framework for multiple agent types.

Qualifications

• 4+ years of experience in data engineering, ML engineering, or software engineering.
• Bachelor's or Master's degree in Computer Science, Machine Learning, or a related quantitative field.
• Strong proficiency in Python.
• Experience building and maintaining production data pipelines.
• Strong SQL skills.
• Experience working with at least one cloud platform (AWS preferred).

Responsibilities

• Data Engineering: Build and maintain ETL pipelines for heterogeneous data sources (traces, logs, transcripts, user feedback).
• Platform & Observability: Develop dashboards and monitoring tools for AI quality metrics.
• AI / ML Evaluation Tooling: Apply and extend LLMasjudge evaluation patterns.
• Collaboration: Partner with data science, engineering, and product teams to define quality criteria.

Skills

Python

SQL

Data pipelines

Cloud platforms (AWS)

Education

BSc/MSc in Computer Science or ML

Tools

LangSmith

Playwright

LLMasjudge

CI/CD

Job description

Judi Health is seeking an AI Evaluation Engineer to build testing frameworks, metrics, and tooling to assess safety, reliability, and accuracy of AI models in production. You will translate product goals into measurable quality targets and collaborate with data science, engineering, and product teams.

Responsibilities include creating evaluation pipelines, dashboards, and tooling, and contributing to a standardized evaluation framework for multiple agent types.