Behavioral Health Specialist - AI Evaluation

Mercor; New York (NY); On-site; USD 83,000 - 124,000; Part time

Job summary

Cincinnatus LLC is placing behavioral health experts to help a leading AI lab evaluate how its models respond in everyday conversations about relationships, wellbeing and beliefs. This part-time role requires 20 hours per week (up to 40) and involves reviewing model outputs, identifying improvements and contributing to balanced, nonjudgmental guidance.

You'll work within a W-2 employment framework, with the opportunity to be placed at a top AI lab as part of the extended workforce, collaborating

Qualifications

• A degree in psychology, counseling, social work, behavioral health, or related field, or equivalent professional experience in mental health.
• 3+ years of professional experience in mental health, counseling or social services; licensure valued but not required.
• Demonstrated ability to remain neutral across diverse perspectives and explain professional judgments.
• Familiarity with sycophancy, cognitive distortions, healthy boundaries and client-centered approaches such as motivational interviewing.
• Available for at least 20 hours/week during weekdays.
• Strong written communication and ability to deliver precise, well-structured feedback.

Responsibilities

• Evaluate conversations between users and AI models for neutrality, appropriateness and safety.
• Identify patterns of concern and document rationale.
• Develop rubrics, guidelines and reference responses for balanced replies.
• Design test scenarios that probe handling of sensitive topics.
• Collaborate with researchers to maintain consistent evaluation standards.

Skills

Neutrality

Written communication

Clear feedback

Time commitment

Education

Degree in psychology or related field

3+ years in mental health or related field

Licensure valued but not required

Job description

Help a leading AI lab ensure its models respond to people with care, balance and sound judgment.

1. Overview

A leading AI lab is seeking behavioral health experts to help evaluate and improve how its AI models handle sensitive, everyday conversations. People increasingly turn to AI for support with relationships, family dynamics, emotional wellbeing, personal beliefs and difficult life decisions. These conversations rarely involve crisis content, but they are exactly where a model's judgment matters most: whether it stays balanced, avoids taking sides, resists telling people what they want to hear, respects a person's beliefs without endorsing or dismissing them, and understands the limits of its role. You'll review these interactions, assess whether the model's responses are appropriate, neutral and safe, and help the lab's researchers define what good looks like. If you bring professional experience in mental health, counseling, social work or behavioral science, and you can articulate clearly why one response serves a person better than another, this role is for you. This is a part-time commitment of at least 20 hours per week, with the option to increase to up to 40 hours per week.

This is a W-2 employment position with Cincinnatus LLC, with the opportunity to be placed at a leading AI lab as part of their extended workforce.

2. Key Responsibilities

• Evaluate conversations between users and AI models across topics such as relationship and family advice, emotional wellbeing, spiritual and metaphysical questions, and unconventional or unfounded beliefs, and assess whether responses are neutral, appropriate and safe.

• Identify patterns of concern, including excessive agreement, taking sides, reinforcing distorted or unfounded beliefs, moralizing, or overstepping into clinical or directive advice, and document them with clear written rationale.

• Develop rubrics, guidelines and reference responses that define a balanced, supportive and appropriately bounded reply, grounded in established practice from counseling and behavioral health.

• Design test scenarios and conversations that probe how models handle sensitive but non crisis topics.

• Collaborate with the lab's researchers and fellow experts to keep evaluation standards consistent, calibrated and well documented.

3. Core Qualifications

• A degree in psychology, counseling, social work, behavioral health, behavioral science, human services or a closely related field, or equivalent professional experience in the mental health field.

• 3+ years of professional experience supporting people in a mental health, counseling or social services setting, for example as a therapist, counselor, clinical social worker, psychologist, psychiatric nurse, case manager, crisis counselor, peer support specialist or mental health advocate. Clinical licensure (for example LMFT, LCSW, LPC, LMHC, PsyD or PhD) is valued but not required.

• Demonstrated ability to remain neutral and nonjudgmental across diverse perspectives, relationships, belief systems and worldviews, and to explain the reasoning behind a professional judgment.

• Working familiarity with concepts such as sycophancy, cognitive distortions, healthy boundaries and client centered approaches such as motivational interviewing.

• Ability to engage reliably for at least 20 hours/week during weekdays.

• Strong written communication skills and the ability to deliver precise, well structured written feedback.

Nice to have: a background in AI safety, applied ethics, trust and safety or content policy; experience in couples, family or relationship counseling; familiarity with spiritual care, religious or alternative belief communities, or the psychology of misinformation and conspiracy belief; prior experience evaluating, annotating or red teaming AI systems.

About Cincinnatus LLC:Cincinnatus LLC is an enterprise staffing company that partners with leading technology companies to source and employ highly skilled professionals for contingent and contract-based opportunities. Cincinnatus serves as the employer of record for these engagements, providing W-2 employment, payroll, benefits, and compliance, while placing employees directly within client teams to work on high-impact initiatives.

Equal Employment Opportunity:Cincinnatus is proud to be an Equal Employment Opportunity employer. We do not discriminate based upon race, religion, color, national origin, sex (including pregnancy, childbirth, reproductive health decisions, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, genetic information, political views or activity, or any other legally protected characteristic.