Data Scientist

Programmers.io; Louisville (KY); Hybrid; USD 90,000 - 130,000; Full time

Job summary

Programmers.io seeks a Data Scientist to build and maintain a robust evaluation framework for conversational AI systems in a hybrid Louisville, KY environment. You will develop automated AI quality measurement solutions, including LLM-as-a-Judge systems, and collaborate with product teams and governance stakeholders to ensure accuracy and safety of AI experiences.

You will design experiments, analyze model performance with statistical best practices, and create dashboards to support Responsible

Qualifications

• Master's degree or PhD in Data Science, Statistics, Computer Science, ML, or related field, or equivalent experience.
• 3+ years of experience in data science, ML, analytics, or AI model evaluation.
• Experience analyzing voice-based language data.
• Experience with Generative AI, LLMs, or conversational AI systems.
• Experience with cloud-based AI platforms and experimentation environments.

Responsibilities

• Design and develop LLM-as-a-Judge evaluation frameworks for conversational AI systems.
• Build automated monitoring solutions to evaluate millions of voice and chat-based interactions.
• Create dashboards and reporting to support governance, compliance, and Responsible AI reviews.
• Measure and report precision, recall, false-positive rates, and false-negative rates.
• Analyze hallucination rates, intent classification accuracy, transcript quality (WER), fairness metrics.
• Collaborate with annotation teams to improve label quality and gold-standard datasets.

Skills

Python

Data science tooling

Experiment design

Statistical analysis

Model evaluation

Communication with stakeholders

Education

Master's or PhD in Data Science/Statistics/CS/ML

Tools

Python libraries

Job description

Location - Louisville, Kentucky ( Hybrid)

Job Description:

We are seeking a Data Scientist to help build and maintain a robust evaluation framework for conversational AI systems. This role will focus on developing automated AI quality measurement solutions, including LLM-as-a-Judge systems, to assess hallucination rates, intent accuracy, transcript quality, and responsible AI metrics at scale.

As a key member of our AI Quality team, you will work closely with annotation specialists, product teams, and governance stakeholders to ensure our AI experiences are accurate, safe, compliant, and customer-centric.

Design and develop LLM-as-a-Judge evaluation frameworks for conversational AI systems.

Build automated monitoring solutions to evaluate millions of voice and chat-based customer interactions.

Collaborate with user experience teams to identify core metrics and evaluation criteria.

Validate model performance against human-labeled datasets using statistical best practices.

Measure and report precision, recall, false-positive rates, and false-negative rates.

Develop statistical sampling methodologies and quality measurement frameworks.

Continuously calibrate and improve evaluation models as production AI systems evolve.

Analyze hallucination rates, intent classification accuracy, transcript quality (WER), guardrail effectiveness, vulnerability to jailbreak attempts, and fairness metrics.

Create dashboards and reporting to support governance, legal, compliance, and Responsible AI reviews.

Collaborate with annotation teams to improve label quality and gold-standard datasets.

Master's degree or PhD in Data Science, Statistics, Computer Science, Machine Learning, or related field, or equivalent experience.

3+ years of experience in data science, machine learning, analytics, or AI model evaluation.

Experience analyzing voice-based language data.

Strong experience with Python and data science tooling.

Experience designing experiments and evaluating machine learning models.

Knowledge of statistical analysis, sampling methodologies, and model validation techniques.

Experience communicating technical findings to both technical and non-technical stakeholders.

Experience with Generative AI, LLMs, or conversational AI systems.

Familiarity with AI safety, Responsible AI, fairness, bias, and governance frameworks.

Experience with cloud-based AI platforms and experimentation environments.