Remote LLM Safety Evaluation Consultant

UNICEF - United Nations Children’s Fund; Valencia; Remote; EUR 65,000 - 111,000; Full time

Job summary

UNICEF - United Nations Children’s Fund seeks an AI Technical Consultant for LLM safety evaluation within the Digital Guardrails program in Valencia, Spain. The role is home-based with a 6-month+ duration, focusing on structured prompt testing, model evaluation, and safety research.

You will design repeatable tests, collect metadata, and collaborate with subject-matter experts to refine testing infrastructure and datasets for review.

Qualifications

• Advanced degree in CS/AI/DS/ML/InfoSec or related field.
• Minimum five years of relevant AI/ML, software engineering or data science experience.
• Hands-on experience evaluating LLM outputs across models and configurations.
• Prompt testing, AI safety testing and/or AI red teaming experience.
• Strong Python skills and experience with LLM APIs.
• Ability to design structured, repeatable tests and capture metadata.
• Understanding of system prompts, content filters and safety controls.
• Proficiency with CSV, JSON and JSONL for datasets.
• Experience deploying open-source LLMs in cloud environments.
• Ability to communicate technical findings clearly to non-technical stakeholders.
• Fluency in English is required.
• Azure AI Foundry or comparable evaluation-platform experience is desirable.

Responsibilities

• Prepare, clean and structure the test prompt set.
• Test prompts across LLM providers and configurations and curate informative responses.
• Evaluate safety configurations and document effects on behavior.
• Capture model responses and test metadata.
• Troubleshoot API, model, safety-filter or platform issues.
• Develop and maintain code to call LLM APIs and deploy/open-source models for evaluation.
• Review state-of-the-art child-safety and AI-safety research for prompts and risks.
• Organize outputs for expert review in a simple format.
• Support rating, ranking and reviewer comments.
• Capture review results in a structured, exportable format.
• Iterate prototype based on expert feedback.
• Engage with reviewers to refine testing infrastructure.
• Support re-testing when new failure modes are identified.
• Translate feedback into new/refined test cases.
• Maintain a versioned prompt-response dataset.

Skills

LLM evaluation

Python

Prompt testing

Open-source LLMs

API troubleshooting

CSV JSON data

Clear communication

Model evaluation

Education

Master’s degree in CS/AI

Tools

Azure AI Foundry

LLM APIs

Cloud deployment

Model serving

Job description

UNICEF - United Nations Children’s Fund seeks an AI Technical Consultant for LLM safety evaluation within the Digital Guardrails program in Valencia, Spain. The role is home-based with a 6-month+ duration, focusing on structured prompt testing, model evaluation, and safety research.

You will design repeatable tests, collect metadata, and collaborate with subject-matter experts to refine testing infrastructure and datasets for review.