AI Evaluation & Quality Engineer

Capitalrx

Charlotte (NC)

On-site

USD 134,800 - 168,500

Full time

14 days+

Get more replies from employers

Send a job-specific resume in minutes.

Job summary

Judi Health is seeking an AI Evaluation Engineer in Charlotte, NC to design and operate robust evaluation frameworks for AI models and autonomous agents. You will partner with scientists and engineers to translate product goals into measurable quality targets and lead end-to-end evaluation from unit tests to online analytics.

The role focuses on data engineering, platform observability, and tooling to support continuous quality and safety benchmarking.

Qualifications

4+ years of data engineering, ML engineering, or software engineering experience.
Bachelor’s or Master’s degree in Computer Science, Machine Learning, or related quantitative field.
Strong proficiency in Python.
Experience building and maintaining production data pipelines.
Strong SQL skills.
Experience with at least one cloud platform (AWS preferred).

Responsibilities

Data Engineering: Build and maintain ETL pipelines for heterogeneous data sources (traces, logs, transcripts, user feedback).
Platform & Observability: Develop dashboards and monitoring tools for AI quality metrics; integrate evaluations into CI/CD pipelines.
AI / ML Evaluation Tooling: Apply and extend LLM‑as‑judge evaluation patterns; design metrics for stochastic systems.
Collaboration: Partner with data science, engineering, and product teams to define good AI behavior in production.