Test and Evaluation Specialist

Profectus; Canberra; On-site; AUD 150,000 - 190,000; Full time

Job summary

Profectus is seeking an experienced AI Test and Evaluation and Assurance Specialist to support the safe adoption of AI-enabled capability within Australian Government contexts from Canberra. The role provides independent, evidence-based assessment of AI-enabled systems, governance artefacts and advisory support for risk and suitability at senior levels.

The successful candidate will plan and oversee AI assurance activities, review technical papers, and translate findings into clear reports and

Qualifications

• Tertiary STEM qualification or equivalent professional experience.
• Minimum five years experience in AI/ML/advanced technology governance or assurance.
• Experience applying systems engineering, risk, assurance and test methods to real-world problems.
• Experience planning or overseeing assurance, verification, validation and evaluation activities.
• Strong understanding of AI/ML systems and ability to assess documentation, performance claims and risks.
• Experience developing governance, policy, standards or risk management in tech settings.
• Ability to interpret policy/governance requirements and translate into practical implementation.
• Experience preparing reports for senior stakeholders and decision makers.
• Experience engaging with cross-functional stakeholder groups.
• Postgraduate qualification in AI/ML/Data Science/Systems Engineering/operational analysis.
• Experience delivering AI/ML/data analytics within Commonwealth contexts.
• Experience with generative AI/LLMs/RAG/CV/autonomy or advanced analytics.
• Experience supporting verification/validation/assurance/certification or accreditation.
• Experience acting as customer representative or technical authority.
• Experience assessing vendor claims and solution effectiveness.
• Experience implementing AI governance and responsible AI approaches.
• Experience with cloud AI platforms (Azure/AWS/GCP).
• Experience with AI/ML tools (TensorFlow, PyTorch, Scikit-learn); proficient in Python.
• Relevant industry certifications in AI/ML/cloud AI, systems engineering.

Responsibilities

• Support AI assurance and test across AI-enabled systems.
• Assess technical evidence, performance claims, limitations and risks.
• Coordinate evaluation activities delivered by industry and researchers.
• Advise senior stakeholders on risk and suitability.
• Develop AI governance artefacts including use cases and model cards.
• Translate technical findings into actionable reports for decision makers.
• Capture and transfer knowledge to support workforce continuity.

Skills

AI governance

Risk management

Stakeholder engagement

Technical reporting

Cloud platforms

Python

Education

Postgraduate AI/Data Science/Systems Engineering

Bachelor’s degree in STEM

Tools

TensorFlow

PyTorch

Scikit-learn

Azure/AWS/Google Cloud

Job description

Profectus is seeking an experienced AI Test and Evaluation and Assurance Specialist to support the safe and responsible adoption of AI-enabled capability within an Australian Government/Commonwealth context. The role provides independent, evidence based assessment of AI-enabled systems, covering performance, security, reliability and operational suitability, alongside the governance and assurance products that support capability, acquisition and operational decisions. The work includes assessing technical evidence and vendor claims, overseeing evaluation activities delivered by third parties, and advising senior stakeholders on risk and suitability.

Key Responsibilities

• Support AI assurance and test, evaluation, verification and validation activities across AI-enabled systems
• Assess technical evidence, performance claims, limitations and risks associated with AI-enabled systems
• Contribute to the planning, coordination and oversight of evaluation activities delivered by industry, research organisations and other Commonwealth groups
• Support the client to act as an informed customer and assurance authority for AI activities delivered by third parties
• Develop, maintain and review AI governance and assurance artefacts, including use cases, system cards and model cards
• Review AI related policies, submissions, technical papers and proposals and advise on their implications
• Support the implementation and continuous improvement of AI governance and assurance arrangements
• Analyse complex technical, operational and policy issues and provide evidence based advice to decision makers
• Translate technical findings into clear and actionable advice, reports and decision support products for senior stakeholders
• Identify risks, issues and opportunities associated with AI-enabled capability and recommend management strategies
• Engage with stakeholders across Commonwealth, industry, academia and government to support AI assurance and capability activities
• Capture and transfer corporate knowledge and lessons learned to support workforce continuity

Required Experience

• Tertiary qualification in engineering, computer science, artificial intelligence, data science, machine learning, systems engineering, mathematics, statistics or another relevant STEM discipline, or equivalent professional experience
• Minimum five years experience developing, implementing, managing, assuring, testing, evaluating or governing AI, machine learning or other advanced technology solutions
• Experience applying engineering, systems engineering, risk, assurance, operational analysis or test and evaluation methods to complex real world problems
• Experience planning, conducting, supporting or overseeing assurance, verification, validation, test and evaluation or experimentation activities
• Sound understanding of AI and machine learning systems, including the ability to assess technical documentation, performance claims, evidence and risks
• Experience developing, reviewing or applying governance, assurance, policy, standards or risk management approaches in a technology, engineering or capability setting
• Ability to interpret policy, governance and regulatory requirements and translate them into practical implementation approaches and advice
• Experience preparing reports, briefs, submissions or advice for senior stakeholders and decision makers
• Experience engaging and collaborating with stakeholder groups across technical, operational and governance functions
• Postgraduate qualification in artificial intelligence, machine learning, data science, systems engineering, test and evaluation or operational analysis
• Experience delivering AI, machine learning, data science or advanced analytics within Commonwealth Government, Commonwealth industry or national security
• Experience with generative AI, large language models, retrieval augmented generation, computer vision, autonomy or advanced analytics
• Experience supporting verification, validation, assurance, certification, accreditation or test and evaluation activities
• Experience acting as a customer representative, technical authority, assurance authority, evaluator or independent verification and validation function
• Experience assessing vendor or supplier technical performance, capability claims and solution effectiveness
• Experience developing or implementing AI governance, responsible AI, risk management, compliance, privacy, security or ethical AI approaches
• Experience with cloud based AI and machine learning platforms such as Microsoft Azure AI, AWS or Google Cloud
• Experience with AI and machine learning tools such as TensorFlow, PyTorch or Scikit-learn, and proficiency in Python or similar languages
• Relevant industry certifications in AI, machine learning, cloud AI services, systems engineering or related disciplines

Security Clearance Requirements

• Must be an Australian Citizen
• Must have an active AGSVA NV1 clearance
• Must comply with ITAR requirements

Working Arrangements

• Onsite

Why Profectus

Profectus partners with Australian Government agencies to deliver trusted ICT capability across complex delivery programs. We take a long term, relationship driven approach, supporting our people across the full lifecycle of their engagement while contributing to outcomes of national importance.