AI Quality & Evaluation Engineer

Capital Rx; Charlotte (NC); On-site; USD 120,000 - 180,000; Full time

Job summary

Judi Health in Charlotte, NC is seeking an AI Evaluation Engineer to build testing frameworks, metrics, and tooling that assess safety, reliability, and accuracy of AI models and autonomous agents in production. You will translate product goals into measurable quality targets and collaborate with data science, engineering, and product teams.

You will own data pipelines, evaluation platforms, and experiment tracking, enabling rapid scenario curation, session reconstruction, dashboards, and CI/CD

Qualifications

• 4+ years of experience in data engineering, ML engineering, or software engineering.
• Strong proficiency in Python and SQL with production data pipelines experience.
• Experience building and maintaining production data pipelines.
• Experience with at least one cloud platform (AWS preferred).
• Familiarity with evaluation tooling and scalable analytics is a plus.

Responsibilities

• Data Engineering: Build and maintain ETL pipelines for heterogeneous data sources (traces, logs, transcripts, user feedback).
• Platform & Observability: Develop dashboards and monitoring tools for AI quality metrics and integrate evals into CI/CD pipelines.
• AI / ML Evaluation Tooling: Apply and extend evaluation patterns; design metrics for stochastic systems; track runs and experiments.
• Collaboration: Partner with data science, engineering, and product teams to define success criteria and ensure usability.

Skills

Python

Data engineering

ML engineering

SQL

AWS

Education

Bachelor's or Master's in CS/ML or related field

Tools

LangSmith

Playwright

Job description

Judi Health in Charlotte, NC is seeking an AI Evaluation Engineer to build testing frameworks, metrics, and tooling that assess safety, reliability, and accuracy of AI models and autonomous agents in production. You will translate product goals into measurable quality targets and collaborate with data science, engineering, and product teams.

You will own data pipelines, evaluation platforms, and experiment tracking, enabling rapid scenario curation, session reconstruction, dashboards, and CI/CD