Behavioral Health Specialist - AI Evaluation

Mercor; New York (NY); On-site; USD 42,000 - 84,000; Part time

Job summary

Cincinnatus LLC is seeking behavioral health experts to evaluate and improve AI model responses to sensitive, non-crisis conversations. You will review interactions, determine neutrality and safety, and help define what good responses look like. This is a part-time, W-2 role with potential extension up to 40 hours weekly.

The role requires a background in psychology or related fields and 3+ years in mental health or social services. Strong written communication is essential.

Qualifications

• Degree in psychology, counseling, social work, behavioral health, or related field.
• 3+ years in mental health, counseling or social services setting.
• Licensure valued but not required.
• Ability to remain neutral and explain professional judgment.
• Familiarity with motivational interviewing.

Responsibilities

• Evaluate conversations between users and AI models for neutrality, safety and appropriateness.
• Identify patterns of concern and document rationale.
• Develop rubrics and reference responses for balanced, bounded replies.
• Design test scenarios to probe handling of sensitive topics.
• Collaborate with researchers to keep evaluation standards consistent.

Skills

Empathy and active listening

Neutral / nonjudgmental approach

Written communication

Critical thinking

Articulating reasoning

Time management

Education

Degree in psychology or related field

Clinical licensure (valued)

Job description

Help a leading AI lab ensure its models respond to people with care, balance and sound judgment.

1. Overview

A leading AI lab is seeking behavioral health experts to help evaluate and improve how its AI models handle sensitive, everyday conversations. People increasingly turn to AI for support with relationships, family dynamics, emotional wellbeing, personal beliefs and difficult life decisions. These conversations rarely involve crisis content, but they are exactly where a model's judgment matters most: whether it stays balanced, avoids taking sides, resists telling people what they want to hear, respects a person's beliefs without endorsing or dismissing them, and understands the limits of its role. You'll review these interactions, assess whether the model's responses are appropriate, neutral and safe, and help the lab's researchers define what good looks like. If you bring professional experience in mental health, counseling, social work or behavioral science, and you can articulate clearly why one response serves a person better than another, this role is for you. This is a part-time commitment of at least 20 hours per week, with the option to increase to up to 40 hours per week.

This is a W-2 employment position with Cincinnatus LLC, with the opportunity to be placed at a leading AI lab as part of their extended workforce.

2. Key Responsibilities

• Evaluate conversations between users and AI models across topics such as relationship and family advice, emotional wellbeing, spiritual and metaphysical questions, and unconventional or unfounded beliefs, and assess whether responses are neutral, appropriate and safe.

• Identify patterns of concern, including excessive agreement, taking sides, reinforcing distorted or unfounded beliefs, moralizing, or overstepping into clinical or directive advice, and document them with clear written rationale.

• Develop rubrics, guidelines and reference responses that define a balanced, supportive and appropriately bounded reply, grounded in established practice from counseling and behavioral health.

• Design test scenarios and conversations that probe how models handle sensitive but non crisis topics.

• Collaborate with the lab's researchers and fellow experts to keep evaluation standards consistent, calibrated and well documented.

3. Core Qualifications

• A degree in psychology, counseling, social work, behavioral health, behavioral science, human services or a closely related field, or equivalent professional experience in the mental health field.

• 3+ years of professional experience supporting people in a mental health, counseling or social services setting, for example as a therapist, counselor, clinical social worker, psychologist, psychiatric nurse, case manager, crisis counselor, peer support specialist or mental health advocate. Clinical licensure (for example LMFT, LCSW, LPC, LMHC, PsyD or PhD) is valued but not required.

• Demonstrated ability to remain neutral and nonjudgmental across diverse perspectives, relationships, belief systems and worldviews, and to explain the reasoning behind a professional judgment.

• Working familiarity with concepts such as sycophancy, cognitive distortions, healthy boundaries and client centered approaches such as motivational interviewing.

• Ability to engage reliably for at least 20 hours/week during weekdays.

• Strong written communication skills and the ability to deliver precise, well structured written feedback.

Nice to have: a background in AI safety, applied ethics, trust and safety or content policy; experience in couples, family or relationship counseling; familiarity with spiritual care, religious or alternative belief communities, or the psychology of misinformation and conspiracy belief; prior experience evaluating, annotating or red teaming AI systems.

About Cincinnatus LLC:Cincinnatus LLC is an enterprise staffing company that partners with leading technology companies to source and employ highly skilled professionals for contingent and contract-based opportunities. Cincinnatus serves as the employer of record for these engagements, providing W-2 employment, payroll, benefits, and compliance, while placing employees directly within client teams to work on high-impact initiatives.

Equal Employment Opportunity:Cincinnatus is proud to be an Equal Employment Opportunity employer. We do not discriminate based upon race, religion, color, national origin, sex (including pregnancy, childbirth, reproductive health decisions, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, genetic information, political views or activity, or any other legally protected characteristic.