Application Security AI Evaluation Expert

OpenTrain AI, Inc.; Asia; Remote; PEN 189,000 - 378,000; Part time

Job summary

OpenTrain AI, Inc. is seeking a security-focused contractor to design and evaluate realistic training assets for AI systems. The work blends application security judgment with coding, testing, documentation, and improvements to evaluation pipelines.

You will build small fixtures with vulnerabilities or secure solutions, craft scenarios around sensitive information, and develop deterministic tests while documenting findings for cross-functional teams.

Qualifications

• Extensive application security or secure software development experience.
• Familiarity with OSWE, LLM security, and AI-powered static analysis.
• Ability to design or evaluate security architecture for traditional and agent-based applications.

Responsibilities

• Design and implement small application fixtures with embedded vulnerabilities or secure solutions.
• Create scenarios involving sensitive information disclosure and excessive agency.
• Develop deterministic tests and security-context manifests to assess application security consistently.
• Build hidden answer keys for evaluating vulnerable and secure code.
• Improve synthetic security tasks to remain realistic and challenging.
• Document technical findings, methods, insights, and practical recommendations.
• Review peer contributions and support agent-based security architecture and evaluation-pipeline improvements.

Skills

Application security

Secure SDLC

Code review

Security architecture

Deterministic testing

Technical documentation

AI/ML security

Tools

OSWE

LLM security

Static analysis tools

Job description

The Work

You will create realistic security fixtures and evaluation assets that help train and assess AI systems. The work combines application security judgment with coding, testing, technical documentation, peer feedback, and improvements to evaluation pipelines.

• Design and implement small application fixtures with embedded vulnerabilities or secure solutions.
• Create scenarios involving sensitive information disclosure and excessive agency.
• Develop deterministic tests and security-context manifests to assess application security consistently.
• Build hidden answer keys for evaluating vulnerable and secure code.
• Improve synthetic security tasks so they remain realistic and challenging.
• Document technical findings, methods, insights, and practical recommendations.
• Review peer contributions and support agent-based security architecture and evaluation-pipeline improvements.

What It Pays and Takes

This is a remote contractor role with flexible scheduling. The listing marks the experience level as entry level, while the requirements call for substantial application security or secure software development experience.

• Pay: $40 to $80 per hour.
• Work style: Remote contractor role with flexible scheduling.
• Language: Fluent English.
• Location: Open to candidates in the United Arab Emirates, Argentina, Austria, Bangladesh, Belgium, Bahrain, Brazil, Canada, Switzerland, Chile, Colombia, Germany, Denmark, Algeria, Egypt, Spain, France, the United Kingdom, Ghana, Greece, Indonesia, India, Italy, Jordan, Japan, Kenya, Kyrgyzstan, Kuwait, Kazakhstan, Lebanon, Sri Lanka, Morocco, and Mexico.
• Location: Also open to candidates in Malaysia, Nigeria, the Netherlands, Oman, Peru, the Philippines, Pakistan, Palestine, Portugal, Qatar, Saudi Arabia, Singapore, Thailand, Tunisia, Taiwan, the United States, Uzbekistan, Vietnam, and South Africa.
• Experience: Extensive application security, secure software development, or comparable CyberSWE experience, ideally at a five-year or greater level.
• Knowledge: Strong knowledge of OSWE, LLM security, AI-powered static analysis, and application security best practices.
• Skills: Ability to design or evaluate security architecture for traditional and agent-based applications, build and test codebases, identify and fix advanced vulnerabilities, and communicate precise technical judgments.
• Useful background: Familiarity with synthetic security fixtures and deterministic testing approaches.
• Prior AI training experience is not required.

About AI Training Work

AI training is the human work behind systems that learn from examples, including code reviews, evaluations, and feedback on model outputs. People with strong technical expertise help make those examples accurate, consistent, and useful.