AI Conversation Evaluation & Context Research Specialist

Gramian Consulting; South Africa; On-site; ZAR 700,000 - 1,200,000; Full time

Job summary

Gramian Consultancy is seeking experienced Subject Matter Experts from psychology, social work, counseling, qualitative research, HCI, and related advisory fields to support AI conversation evaluation research.

You will review conversations between people and AI models, assess whether the AI gathered sufficient context before making recommendations, identify missed opportunities for clarification, and provide structured feedback to improve conversational quality.

Qualifications

• 3–5 years of direct professional experience in one or more of the listed domains.
• Licensed Social Work credentials or equivalent.
• Advanced degrees in psychology, social work, or related advisory fields.
• Experience in counseling, interviewing, or client-facing professional work.
• Ability to analyze conversations systematically and identify missing context.

Responsibilities

• Review AI conversations involving interpersonal, workplace, education, family, and wellbeing topics.
• Evaluate whether the AI gathered sufficient context before providing advice.
• Identify missing information and suggest follow-up questions.
• Assess whether additional context would change recommendations.
• Document findings using structured evaluation guidelines.
• Collaborate with research teams to maintain evaluation standards.
• Complete tasks accurately within project timelines.

Skills

Analytical thinking

Interviewing

Qualitative research

Conversation analysis

Structured feedback

Education

MSW or equivalent

LCSW/LICSW/LMSW or equivalent

LMFT/LPC/LCPC or equivalent

PhD or PsyD in Clinical/Counseling/School or IO Psychology

MA/PhD in HCI, Communication, Anthropology, Sociology or Qualitative Research

Job description

About Gramian

Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.

About The Role

We are seeking experienced Subject Matter Experts (SMEs) from psychology, social work, counseling, qualitative research, human-computer interaction (HCI), communication, conflict resolution, coaching, and related advisory fields to support AI conversation evaluation research. You will review conversations between people and AI models, assess whether the AI gathered sufficient context before making recommendations, identify missed opportunities for clarification, and provide structured feedback to improve conversational quality. The ideal candidate has at least 3-5 years of relevant professional experience, strong analytical and interviewing skills, and the ability to distinguish essential, decision‑changing information from details that are less relevant to a recommendation.

Key Responsibilities

• Review AI conversations involving interpersonal relationships, workplace situations, education, family matters, everyday planning, wellbeing, and other real-world decisions
• Evaluate whether the AI gathered sufficient context before providing advice, recommendations, or proposed solutions
• Identify missing information and suggest relevant follow‑up questions that could improve the quality of the AI's response
• Assess whether additional context would materially change the recommendation or decision
• Distinguish between load-bearing information that could change the outcome and non-essential or merely descriptive details
• Conduct qualitative reviews using structured evaluation guidelines and document findings consistently
• Identify missed opportunities for clarification, contextual understanding, and information gathering
• Consider relevant personal, psychological, social, financial, practical, and organizational factors when evaluating conversations
• Collaborate with research teams to maintain consistent evaluation standards and high-quality assessments
• Complete assigned tasks accurately within the required project timelines

Requirements

Candidates must have at least 3-5 years of direct professional experience in one or more of the following domains.

2. Counseling, Therapy, or Psychology

3. Qualitative Research, HCI, or Communication

4. Mediation, Coaching, Student Services, Organizational Consulting, or Related Advisory Fields

• Licensed Social Work
• Master's degree in Social Work (MSW) or an equivalent graduate‑level qualification
• LCSW, LICSW, LMSW, or an equivalent applicable social work credential or license
• Experience in client assessment, case review, interviewing, counseling, or related social‑work practice
• Master's degree in a relevant discipline with an LMFT, LPC, LCPC, or equivalent counseling or therapy credential; or
• PhD or PsyD in Clinical, Counseling, Social, or Industrial‑Organizational Psychology
• Relevant experience in counseling, therapy, psychological assessment, interviewing, case review, or other client‑facing professional work
• MA, PhD, or equivalent advanced expertise in Human‑Computer Interaction, Communication, Anthropology, Sociology, Qualitative Research, or a closely related discipline
• Experience conducting qualitative research, interviews, user research, conversation analysis, thematic analysis, qualitative coding, or similar research activities
• Ability to analyze conversations systematically and identify missing context or patterns in human communication
• Relevant professional qualification, credential, or demonstrated subject‑matter expertise
• At least 5 years of direct advisory or client‑facing experience for candidates qualifying through this pathway
• Experience in coaching, mediation, student advising, organizational consulting, or comparable professional work
• Demonstrated ability to gather context, understand individual or organizational needs, and provide informed recommendations