AI Response Evaluation Analyst

Gramian Consulting Group; United States; Remote; USD 70,000 - 95,000; Full time

Job summary

Gramian Consultancy seeks a detail-oriented Business Analyst – AI Evaluation Specialist to support a project evaluating chatbot outputs and assessing them against business requirements. You will analyze processes, document criteria, and deliver structured assessments to improve AI performance.

The role involves creating reports, collaborating with IT and business stakeholders, and guiding testing and UAT. Strong analytical and communication skills are essential.

Qualifications

• Gather and document business requirements, stakeholder expectations, and project objectives.
• Evaluate chatbot outputs against defined evaluation criteria for accuracy and relevance.
• Create reports, dashboards, and process maps to support findings.
• Translate business requirements into functional specifications and criteria.
• Analyze workflows and data to identify gaps and improvements.
• Maintain consistent evaluation quality and meet deadlines.
• Communicate findings with evidence-based conclusions.

Responsibilities

• Gather, analyze, and document business requirements and project objectives.
• Compare chatbot responses and assess accuracy, relevance, and completeness.
• Conduct structured evaluations and provide evidence-based assessments.
• Translate requirements into functional specs and evaluation criteria.
• Analyze processes to identify gaps and improvement opportunities.
• Create reports, dashboards, and project documentation.
• Support planning, testing, and user acceptance testing (UAT).
• Collaborate with technical teams and stakeholders to ensure requirements are met.
• Track progress, risks, dependencies, and KPIs.
• Present findings and recommendations to support decisions.

Skills

Analytical thinking

Problem solving

Decision making

Attention to detail

Written communication

Documentation

Cross-functional collaboration

Time management

AI evaluation awareness

Education

Bachelor’s degree in Business Administration, Finance, Computer Science, or related field

Tools

Excel

SQL

Power BI

Tableau

Job description

Gramian Consultancy seeks a detail-oriented Business Analyst – AI Evaluation Specialist to support a project evaluating chatbot outputs and assessing them against business requirements. You will analyze processes, document criteria, and deliver structured assessments to improve AI performance.

The role involves creating reports, collaborating with IT and business stakeholders, and guiding testing and UAT. Strong analytical and communication skills are essential.