AI Safety Data Scientist — Remote Risk & Evaluation

Aquent; New York (NY); On-site; USD 120,000 - 160,000; Full time

Job summary

Aquent is seeking an experienced Data Scientist to measure and monitor risks in conversational and agentic AI products. You will investigate emerging safety risks in production and develop scalable measurement, evaluation, and monitoring systems for policy and product improvements.

You will collaborate with Product, Engineering, and Trust & Safety teams, translate insights into mitigations, and communicate findings to stakeholders with clarity and impact.

Qualifications

• Experience delivering safety evaluations for AI products.
• Ability to design evaluation datasets and rubrics.
• Strong written and verbal communication for stakeholder uptake.
• Comfort structuring ambiguous problems and defining success criteria.

Responsibilities

• Develop and scale risk monitoring and safety measurement systems across AI features.
• Design safety metrics and evaluation frameworks for production systems (false positives/negatives, safety risk prevalence).
• Inspect, debug, and use Python data analysis scripts and SQL pipelines (LLM tools optional).
• Calibrate LLM-evaluation workflows, label datasets, and build dashboards for leadership.
• Collaborate with Product, Engineering, and Trust & Safety to translate insights into mitigations and policies.
• Communicate analytical findings clearly to technical and non-technical stakeholders.

Skills

Python

SQL

Data analysis

AI safety

Risk metrics

Tools

Claude

Codex

Job description

Aquent is seeking an experienced Data Scientist to measure and monitor risks in conversational and agentic AI products. You will investigate emerging safety risks in production and develop scalable measurement, evaluation, and monitoring systems for policy and product improvements.

You will collaborate with Product, Engineering, and Trust & Safety teams, translate insights into mitigations, and communicate findings to stakeholders with clarity and impact.