AI Assurance Engineer — Mission-Critical Evaluations

Johns Hopkins Applied Physics Lab; Laurel (MD); On-site; USD 100,000 - 245,000; Full time

Job summary

Johns Hopkins Applied Physics Laboratory seeks an AI Assurance Engineer to design and execute evaluations of generative AI models and agentic systems for defense and health missions. You will develop test strategies, define success criteria, and apply benchmarking frameworks to measure performance, safety, and reliability.

You will collaborate with AI researchers, engineers, and mission partners, analyze results, and communicate recommendations to sponsors.

Qualifications

• Bachelor's degree in CS, data science, AI, ML, statistics, cybersecurity, or related field.
• Experience with Python and AI software development.
• Familiarity with generative AI models, agentic systems, or ML concepts.
• Knowledge of test and evaluation principles and ability to analyze results.
• Strong written and verbal communication and teamwork.
• Eligibility for Interim Secret clearance with potential Top-Secret clearance; U.S. citizenship required.

Responsibilities

• Design and execute assurance evaluations for generative AI models and agentic systems in defense and healthcare missions.
• Develop test strategies, evaluation plans, test cases, and success criteria.
• Apply AI evaluation methods, benchmarks, and test frameworks to identify risks.
• Collaborate with AI researchers, engineers, and mission partners to advance assurance capabilities.
• Analyze evaluation results and communicate findings, recommendations, and decisions to sponsors.
• Evaluate AI capabilities in realistic mission environments to inform risk-based deployment decisions.

Skills

Python

Test & evaluation

Communication skills

Collaboration

Security clearance eligibility

Education

Bachelor's degree in Computer Science or related field

Master's degree in related field

Job description

Johns Hopkins Applied Physics Laboratory seeks an AI Assurance Engineer to design and execute evaluations of generative AI models and agentic systems for defense and health missions. You will develop test strategies, define success criteria, and apply benchmarking frameworks to measure performance, safety, and reliability.

You will collaborate with AI researchers, engineers, and mission partners, analyze results, and communicate recommendations to sponsors.