AI Quality Scientist - LLM Evaluation & Metrics (Hybrid)

Programmers.io; Louisville (KY); Hybrid; USD 90,000 - 130,000; Full time

Job summary

Programmers.io seeks a Data Scientist to build and maintain a robust evaluation framework for conversational AI systems in a hybrid Louisville, KY environment. You will develop automated AI quality measurement solutions, including LLM-as-a-Judge systems, and collaborate with product teams and governance stakeholders to ensure accuracy and safety of AI experiences.

You will design experiments, analyze model performance with statistical best practices, and create dashboards to support Responsible

Qualifications

• Master's degree or PhD in Data Science, Statistics, Computer Science, ML, or related field, or equivalent experience.
• 3+ years of experience in data science, ML, analytics, or AI model evaluation.
• Experience analyzing voice-based language data.
• Experience with Generative AI, LLMs, or conversational AI systems.
• Experience with cloud-based AI platforms and experimentation environments.

Responsibilities

• Design and develop LLM-as-a-Judge evaluation frameworks for conversational AI systems.
• Build automated monitoring solutions to evaluate millions of voice and chat-based interactions.
• Create dashboards and reporting to support governance, compliance, and Responsible AI reviews.
• Measure and report precision, recall, false-positive rates, and false-negative rates.
• Analyze hallucination rates, intent classification accuracy, transcript quality (WER), fairness metrics.
• Collaborate with annotation teams to improve label quality and gold-standard datasets.

Skills

Python

Data science tooling

Experiment design

Statistical analysis

Model evaluation

Communication with stakeholders

Education

Master's or PhD in Data Science/Statistics/CS/ML

Tools

Python libraries

Job description

Programmers.io seeks a Data Scientist to build and maintain a robust evaluation framework for conversational AI systems in a hybrid Louisville, KY environment. You will develop automated AI quality measurement solutions, including LLM-as-a-Judge systems, and collaborate with product teams and governance stakeholders to ensure accuracy and safety of AI experiences.

You will design experiments, analyze model performance with statistical best practices, and create dashboards to support Responsible