Senior Applied Scientist - AI Red Teaming & Model Risk

SupportFinity™; New York (NY); On-site; USD 190,000 - 211,000; Full time

Benefits offered by this job

401(k) plan

Bonus program

Equity opportunity

Job summary

Uber is seeking a Senior Applied Scientist to join the AI Red Teaming efforts, focusing on adversarial evaluation, failure analysis, and risk discovery in AI models and agents.

You will design experiments and evaluation frameworks to surface unsafe behaviors, including prompt injection, jailbreaking, and memory poisoning. This role emphasizes safety, robustness, and rigorous scientific methods in real-world AI systems.

Qualifications

• 5+ years of experience as a Data Scientist, Applied Scientist, or ML Scientist.
• Hands-on experience with LLMs or generative AI systems.
• Direct experience with AI red teaming, model safety, or adversarial evaluation.
• Direct experience with prompt injection, jailbreaks, and LLM failure modes.
• Strong background in experimental design, evaluation, and statistical analysis.
• Experience analyzing complex model behavior and failure cases beyond standard metrics.
• Proficiency in Python and common DS/ML tooling.

Responsibilities

• Design and execute AI red-teaming experiments against LLMs and AI agents to identify: prompt injection, jailbreaking, policy bypass, model and tool poisoning, context/memory poisoning, behavioral drift, unsafe autonomy.
• Develop adversarial datasets, probes, and test harnesses to systematically evaluate model and agent behavior under attack.
• Define and track AI risk metrics beyond accuracy (e.g., failure rates, drift indicators, unsafe action likelihood, confidence miscalibration).
• Analyze agent workflows and decision traces to understand how failures emerge across multi-step reasoning and tool use.
• Collaborate with security engineers and AI platform teams to translate findings into guardrails, mitigations, and design improvements.
• Build reusable evaluation pipelines to support continuous red teaming and regression testing as models and agents evolve.

Skills

5+ years experience

LLMs / Generative AI

AI red teaming

Prompt injection / jailbreaks

Experimental design

Python

Job description

Uber |

Uber | Posted Feb 15

Full-time

New York

Unknown

About The Role

As AI systems-particularly LLMs and agentic AI-become core to our products and internal platforms, understanding how these systems fail is just as important as improving their performance. We are looking for a Senior Applied Scientist to join our AI Red Teaming efforts and focus on adversarial evaluation, failure analysis, and risk discovery in AI models and AI agents.

In this role, you will systematically probe AI systems to uncover unsafe, unintended, or harmful behaviors, including prompt injection, jailbreaks, behavioral drift, tool misuse, and context or memory poisoning. You will design experiments, build evaluation frameworks, and analyze outcomes to surface risks that traditional ML metrics do not capture.

This role is ideal for a data scientist who enjoys working at the edge of model behavior, cares deeply about safety and robustness, and wants to apply scientific rigor to securing real-world AI systems.

What The Candidate Will Need / Bonus Points

---- What the Candidate Will Do ----

• Design and execute AI red-teaming experiments against LLMs and AI agents to identify: prompt injection (direct & indirect), jailbreaking and policy bypass, model and tool poisoning, context and memory poisoning, behavioral drift and unsafe autonomy
• Develop adversarial datasets, probes, and test harnesses to systematically evaluate model and agent behavior under attack.
• Define and track AI risk metrics beyond accuracy (e.g., failure rates, drift indicators, unsafe action likelihood, confidence miscalibration).
• Analyze agent workflows and decision traces to understand how failures emerge across multi-step reasoning and tool use.
• Collaborate with security engineers and AI platform teams to translate findings into guardrails, mitigations, and design improvements.
• Build reusable evaluation pipelines to support continuous red teaming and regression testing as models and agents evolve.

Basic Qualifications

• 5+ years of experience as a Data Scientist, Applied Scientist, or ML Scientist.
• Hands-on experience working with LLMs or generative AI systems.
• Direct experience with AI red teaming, model safety, or adversarial evaluation.
• Direct experience with prompt injection, jailbreaks, and LLM failure modes.
• Strong background in experimental design, evaluation, and statistical analysis.
• Experience analyzing complex model behavior and failure cases beyond standard metrics.
• Proficiency in Python and common DS/ML tooling.

Preferred Qualifications

• Experience evaluating agentic systems, including tool use, memory, or multi-step workflows.
• Knowledge of GenAI architectures (transformers, embeddings, RAG, agent frameworks).
• Experience building custom evaluation datasets or simulation environments.
• Background or strong interest in security, privacy, or trust & safety.
• Familiarity with AI evaluation tools (e.g., custom judges, LLM-as-judge, simulation frameworks).

For New York, NY-based roles: The base salary range for this role is USD$190,000 per year - USD$211,000 per year. For San Francisco, CA-based roles: The base salary range for this role is USD$190,000 per year - USD$211,000 per year. For Seattle, WA-based roles: The base salary range for this role is USD$190,000 per year - USD$211,000 per year. For Sunnyvale, CA-based roles: The base salary range for this role is USD$190,000 per year - USD$211,000 per year. For all US locations, you will be eligible to participate in Uber's bonus program, and may be offered an equity award & other types of comp. All full-time employees are eligible to participate in a 401(k) plan. You will also be eligible for various benefits. More details can be found at the following link https://jobs.uber.com/en/benefits., For New York, NY-based roles: The base salary range for this role is USD$190,000 per year - USD$211,000 per year. For San Francisco, CA-based roles: The base salary range for this role is USD$190,000 per year - USD$211,000 per year. For Seattle, WA-based roles: The base salary range for this role is USD$190,000 per year - USD$211,000 per year. For Sunnyvale, CA-based roles: The base salary range for this role is USD$190,000 per year - USD$211,000 per year. For all US locations, you will be eligible to participate in Uber's bonus program, and may be offered an equity award & other types of comp. All full-time employees are eligible to participate in a 401(k) plan. You will also be eligible for various benefits. More details can be found at the following link https://jobs.uber.com/en/benefits.

About the company

Uber