AI Evaluation Subject Matter Expert Charleston, SC

Foxhole Technology, Llc; Charleston (SC); On-site; USD 90,000 - 130,000; Full time

Job summary

Foxhole Technology, Llc is looking for an AI Evaluation SME to support the Navy’s Next Generation CANES environment. This role ensures AI capabilities are mission-relevant and suitable for deployment. Key responsibilities include developing evaluation frameworks and assessment methods while collaborating across various teams.

Ideal candidates will hold a degree in a technical field and possess experience in AI evaluation and cybersecurity. This position offers an opportunity to impact network operations and cyber defense in a critical environment.

Qualifications

• Bachelor’s degree in a relevant field preferred; advanced degree preferred.
• Experience in evaluating AI, machine learning, cybersecurity, or mission systems.
• Strong understanding of AI evaluation methods and performance metrics.

Responsibilities

• Serve as a senior technical advisor for AI evaluation and test planning.
• Develop AI evaluation strategies and performance assessment methods.
• Assess operational relevance and technical maturity of AI capabilities.

Skills

AI evaluation methods

Test design

Performance metrics

Cybersecurity

Machine learning

Data analytics

Education

Bachelor’s degree in computer science, data science, or related

Advanced degree preferred

Tools

Data analysis tools

Test automation

Job description

Clearance: Active Secret w/TS Capability

Foxhole Technology provides robust cybersecurity and IT support capabilities for federal civilian and defense agencies. A recognized leader in navigating technology and security challenges, Foxhole delivers mission-focused innovations to answer evolving and complex needs. Our talented employee-owners provide agile, scalable services and solutions that solve operational gaps, operate critical systems, and protect and secure the enterprise – across the organization and around the world.

Foxhole Technology is seeking an AI Evaluation SME to join an existing program. The AI Evaluation SME will support the assessment, testing, validation, and operational evaluation of artificial intelligence, machine learning, automation, analytics, and decision-support capabilities being considered for or integrated into the Navy’s Next Generation CANES environment. This role will help ensure AI-enabled capabilities are mission-relevant, reliable, secure, explainable, measurable, and suitable for deployment within afloat, tactical, disconnected, intermittent, limited-bandwidth, and multi-security-domain environments.

The SME will develop evaluation frameworks, test methods, metrics, datasets, scenarios, risk assessments, and reporting products that help Navy and CACI stakeholders determine whether AI-enabled capabilities improve network operations, cyber defense, system administration, predictive maintenance, anomaly detection, configuration management, mission planning, or other CANES-related functions.

Key Responsibilities

• Serve as a senior technical advisor for AI evaluation, test planning, performance assessment, and operational suitability analysis in support of Next Generation CANES modernization.
• Develop AI evaluation strategies, test plans, measures of effectiveness, measures of performance, success criteria, risk indicators, and evaluation scorecards.
• Assess AI, machine learning, generative AI, automation, analytics, and decision-support capabilities for operational relevance, technical maturity, cyber risk, reliability, maintainability, explainability, human oversight, and fleet suitability.
• Evaluate AI-enabled tools for use cases such as network monitoring, cyber anomaly detection, event correlation, predictive maintenance, help desk automation, configuration compliance, system health monitoring, log analysis, vulnerability prioritization, and operational decision support.
• Design test scenarios that reflect Navy afloat operating conditions, including limited bandwidth, disconnected operations, contested cyber environments, cross-domain constraints, variable data quality, and platform-specific operational limitations.
• Define data requirements, ground truth methods, evaluation datasets, labeling approaches, validation methods, and performance baselines for AI-enabled capabilities.
• Assess AI model performance using appropriate metrics such as accuracy, precision, recall, false positive rate, false negative rate, latency, robustness, drift, confidence calibration, explainability, and operational impact.
• Evaluate risks associated with hallucination, model brittleness, adversarial manipulation, data poisoning, prompt injection, bias, over-reliance, model drift, cybersecurity exposure, and failure modes in operational environments.
• Support AI red teaming, cyber survivability assessment, adversarial testing, safety reviews, and responsible AI evaluation activities.
• Develop human-machine teaming concepts, operator-in-the-loop workflows, trust calibration approaches, escalation procedures, and recommended guardrails for AI-enabled tools.
• Produce technical reports, evaluation findings, executive summaries, test observations, data analysis products, and recommendations for Navy and CACI leadership.
• Collaborate with systems engineers, cybersecurity engineers, software developers, data scientists, network engineers, operational testers, fleet users, and government stakeholders.
• Support technical interchange meetings, design reviews, test readiness reviews, operational assessments, demonstrations, and acquisition decision support.
• Provide SME input on AI governance, responsible AI implementation, model lifecycle management, configuration control, sustainment, monitoring, and continuous evaluation.

Required Qualifications

• Bachelor’s degree in computer science, data science, artificial intelligence, engineering, mathematics, statistics, cybersecurity, operations research, information systems, or a related technical discipline preferred. Advanced degree preferred.
• Additional years of directly relevant AI evaluation, test, cybersecurity, Navy, or DoD mission system experience may be considered in lieu of a degree.
• Demonstrated experience evaluating AI, machine learning, data analytics, automation, or decision-support systems in defense, intelligence, cybersecurity, network operations, enterprise IT, or mission system environments.
• Strong understanding of AI / ML evaluation methods, test design, performance metrics, validation approaches, model limitations, and operational risk assessment.
• Experience developing test plans, evaluation frameworks, measures of effectiveness, measures of performance, data collection plans, and technical reports.
• Familiarity with cybersecurity, enterprise networks, tactical networks, system monitoring, anomaly detection, log analytics, or network operations use cases.
• Ability to assess AI-enabled systems in operationally constrained environments, including limited bandwidth, degraded connectivity, edge computing, and mission-critical infrastructure.
• Understanding of responsible AI concepts, including transparency, explainability, human oversight, robustness, security, bias, accountability, and lifecycle monitoring.
• Experience working with cross-functional engineering, cyber, data science, software, test, and government stakeholder teams.
• Strong written and verbal communication skills, including the ability to brief complex AI evaluation findings to technical and non-technical audiences.

Desired Qualifications

• Experience supporting Navy, DoD, tactical edge, afloat, C4I, cyber, enterprise IT, or mission command systems.
• Familiarity with CANES, Navy afloat networks, ADNS, NAVWAR programs, RMF, cyber survivability testing, operational test, developmental test, or fleet experimentation.
• Experience evaluating generative AI, large language models, retrieval-augmented generation, autonomous agents, AI-assisted cyber tools, AI-enabled network operations, or predictive analytics systems.
• Knowledge of DoD responsible AI guidance, NIST AI Risk Management Framework concepts, RMF, Zero Trust, DevSecOps, MLOps, model monitoring, or secure software supply chain practices.
• Experience with data analysis tools, scripting, statistical evaluation, dashboards, test automation, or model performance analysis.
• Experience with AI red teaming, adversarial ML, cyber test events, operational assessments, or acquisition decision support.
• Top Secret clearance or SCI eligibility.

Foxhole Technology is an Equal Opportunity Employer and makes hiring decisions without regard to race, color, religion, sex (including pregnancy, childbirth and sexual orientation), national origin, age, disability, genetic information, military/veteran status, or any other protected class.