Application Security AI Evaluation Expert

OpenTrain AI, Inc.; United States; Remote; USD 55,000 - 110,000; Part time

Job summary

OpenTrain AI, Inc. is seeking an Application Security Evaluation Expert to build secure and vulnerable application fixtures and deterministic tests for AI systems. This remote contractor role emphasizes strong security judgment, coding, testing, and documentation.

You will design fixtures with embedded vulnerabilities or secure solutions, develop deterministic tests, document findings, and support improvements to evaluation pipelines for robust AI training.

Qualifications

• Extensive application security experience or secure software development experience.
• Knowledge of OSWE, LLM security, and AI-powered static analysis.
• Ability to design or evaluate security architecture for traditional and agent-based applications.

Responsibilities

• Design and implement small application fixtures with embedded vulnerabilities or secure solutions.
• Create scenarios involving sensitive information disclosure and excessive agency.
• Develop deterministic tests and security-context manifests to assess application security consistently.
• Build hidden answer keys for evaluating vulnerable and secure code.
• Improve synthetic security tasks so they remain realistic and challenging.
• Document technical findings, methods, insights, and practical recommendations.
• Review peer contributions and support agent-based security architecture and evaluation-pipeline improvements.

Skills

Application security

Secure software development

OSWE knowledge

LLM security

AI-powered static analysis

Job description

Application Security AI Evaluation Expert

Build secure and vulnerable application fixtures, deterministic tests, and evaluation assets for AI systems. This remote contractor role pays $40 to $80 per hour, offers flexible scheduling, and requires strong application security skills.

Remote Hourly · $40–$80/hr

$40–$80/hr

Compensation

Contract, part-time

Engagement

Remote

Location

The Work

You will create realistic security fixtures and evaluation assets that help train and assess AI systems. The work combines application security judgment with coding, testing, technical documentation, peer feedback, and improvements to evaluation pipelines.

• Design and implement small application fixtures with embedded vulnerabilities or secure solutions.


• Create scenarios involving sensitive information disclosure and excessive agency.


• Develop deterministic tests and security-context manifests to assess application security consistently.


• Build hidden answer keys for evaluating vulnerable and secure code.


• Improve synthetic security tasks so they remain realistic and challenging.


• Document technical findings, methods, insights, and practical recommendations.


• Review peer contributions and support agent-based security architecture and evaluation-pipeline improvements.

What It Pays and Takes

This is a remote contractor role with flexible scheduling. The listing marks the experience level as entry level, while the requirements call for substantial application security or secure software development experience.

• Pay: $40 to $80 per hour.


• Work style: Remote contractor role with flexible scheduling.


• Location: Open to candidates in the United Arab Emirates, Argentina, Austria, Bangladesh, Belgium, Bahrain, Brazil, Canada, Switzerland, Chile, Colombia, Germany, Denmark, Algeria, Egypt, Spain, France, the United Kingdom, Ghana, Greece, Indonesia, India, Italy, Jordan, Japan, Kenya, Kyrgyzstan, Kuwait, Kazakhstan, Lebanon, Sri Lanka, Morocco, and Mexico.


• Location: Also open to candidates in Malaysia, Nigeria, the Netherlands, Oman, Peru, the Philippines, Pakistan, Palestine, Portugal, Qatar, Saudi Arabia, Singapore, Thailand, Tunisia, Taiwan, the United States, Uzbekistan, Vietnam, and South Africa.


• Experience: Extensive application security, secure software development, or comparable CyberSWE experience, ideally at a five-year or greater level.


• Knowledge: Strong knowledge of OSWE, LLM security, AI-powered static analysis, and application security best practices.


• Skills: Ability to design or evaluate security architecture for traditional and agent-based applications, build and test codebases, identify and fix advanced vulnerabilities, and communicate precise technical judgments.


• Useful background: Familiarity with synthetic security fixtures and deterministic testing approaches.


• Prior AI training experience is not required.

About AI Training Work

AI training is the human work behind systems that learn from examples, including code reviews, evaluations, and feedback on model outputs. People with strong technical expertise help make those examples accurate, consistent, and useful.

Requirements

• Experience: Entry level