Trustworthy AI Senior Scientist, Centre for Frontier AI Research, IAIC

A*STAR - Agency for Science, Technology and Research; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

A*STAR - Agency for Science, Technology and Research's Trustworthy AI team at CFAR, IAIC, seeks motivated scientists to research the safety of generative AI and multi-agent systems. Working with regulators and clinicians, the team develops evidence, testing tools and guidance for real-world deployment.

GenAI and multi-agent systems, such as clinical chatbots built on LLMs and VLMs interpreting medical images, pose risks not covered by current regulations.

Qualifications

• PhD level candidate with strong research track record.
• Proficient in Python and PyTorch with hands-on DL experience.
• Experience in AI safety, adversarial ML or trustworthy ML is desirable.
• Excellent communication and ability to work with clinicians, regulators and engineers.

Responsibilities

• Design and conduct systematic red-teaming and risk assessments of GenAI and multi-agent systems.
• Develop guardrails and risk-mitigation methods, including detection and flagging of high-risk outputs.
• Collaborate with regulatory and clinical partners to translate findings into regulatory-grade tools and guidance.

Skills

Python

PyTorch

AI safety

Adversarial ML

Trustworthy ML

LLMs

VLMs

Multi-agent systems

Communication

Teamwork

Education

PhD in Computer Science / Electrical & Electronic Engineering / Computer Engineering / Biomedical Engineering or related field

Tools

PyTorch

Job description

Competencies

• Integrity
• Critical Thinking
• Innovativeness
• Effective Communication
• Working with Others
• Achieving Outcomes

IDARE Behaviours - 2. Dream & Act Boldly

IDARE Behaviours - 4. Reach Outward, Tackle Real-World Challenges

Short Description

The Trustworthy AI team at the Centre for Frontier AI Research (CFAR), IAIC, is seeking motivated scientists to conduct research on the safety of generative AI (GenAI) and multi-agent systems. Working with regulatory and clinical partners, the team develops the scientific evidence, testing tools and guidance needed for the safe and responsible deployment of AI solutions in real-world.

GenAI and multi-agent systems, such as clinical chatbots built on large language models (LLMs) and vision-language models (VLMs) that interpret medical images, introduce risks that existing regulatory frameworks were not designed to address. These include open-ended outputs, hallucinations, the lack of accepted safety metrics, and vulnerability to jailbreaks, prompt injection and data leakage. You will conduct cutting-edge R&D to assess, test and mitigate these risks, and translate your findings into regulatory-grade toolkits and guidance with real-world meaningful impact.

Key Responsibilities

• Design and conduct systematic red-teaming and risk assessments of GenAI and multi-agent systems, covering adversarial attacks, hallucinations, privacy leakage and performance across diverse scenarios
• Develop efficient guardrails and risk-mitigation methods, such as knowledge validation, hallucination and uncertainty detection, adversarial input detection, and automated flagging of high-risk outputs for human review
• Develop methods for continuous post-deployment monitoring, including near real-time performance tracking, drift detection and anomaly alerting

Long Description

Research areas (one or more of the following)

• Red-teaming and adversarial robustness of LLMs, VLMs and LLM-based agents
• Hallucination detection and mitigation, uncertainty quantification and calibration for medical GenAI
• Guardrails and runtime safety mechanisms for clinical conversational agents and multimodal assistants
• Robustness to distribution shift, and fairness across patient populations, clinical sites and imaging devices
• Safety evaluation metrics and benchmarks, post-deployment performance monitoring and drift detection

Requirements

• PhD degree in Computer Science, Electrical & Electronic Engineering, Computer Engineering, Biomedical Engineering or a related field
• Solid programming skills in Python and hands-on experience with deep learning frameworks such as PyTorch
• Research experience in one or more of the following: AI safety and security, adversarial machine learning, trustworthy machine learning, LLMs or VLMs, and multi-agent systems
• Good communication and teamwork skills, and the ability to work with clinicians, regulators and engineers in a multidisciplinary setting