Quality Assurance Engineer

AI71; Abu Dhabi; On-site; AED 240,000 - 360,000; Full time

Benefits offered by this job

Full Relocation Support

Tax-Free Compensation

Job summary

AI71 is seeking a Senior QA Automation Engineer to define how good AI systems must be. You will bridge Agile development and Systems Engineering, building automated testing that validates LLMs and predictive engines against ground truth datasets.

You will lead TRR and FCA readiness, ensuring data integrity across critical enterprise systems and governance artifacts. The role emphasizes non-deterministic testing, regression suites, performance benchmarks, and secure API validation in a

Qualifications

• 5+ years in QA Automation with data-driven/AI agents
• Experience with ML/AI validation in regulated domains desired
• Ability to define probabilistic pass/fail criteria and confidence levels

Responsibilities

• Lead validation/verification strategies for AI systems
• Build automated testing frameworks to validate AI against Ground Truth
• Ensure TRR and FCA readiness and generate digital evidence
• Validate data consistency between AI agents and critical systems (ERP, BOM)
• Design performance tests for latency and throughput using Locust/K6
• Automate API security validation and RBAC checks

Skills

Python (Pytest)

Selenium/Playwright

Requests

DeepEval

TruLens

Locust

K6

Great Expectations

GitLab CI/CD

QE practices (Shift Left)

Test Pyramid

Mono-repo

Tools

Selenium

Playwright

Requests

DeepEval

TruLens

Locust

K6

Great Expectations

GitLab CI/CD

Job description

Job Description:

The Mission

AI71 is seeking a Senior QA Automation Engineer to lead the validation and verification strategies for ai71's AI transformation. In this role, you will define "what good looks like" for non-deterministic AI systems, ensuring that Large Language Models (LLMs) and predictive engines meet the strict reliability standards required for the defense and enterprise sectors.

You will act as the bridge between Agile development and formal Systems Engineering. Your mandate is to build automated testing frameworks that validate AI behaviors against "Ground Truth" datasets and ensure our AI agents pass rigorous Test Readiness Reviews (TRR) and Functional Configuration Audits (FCA).

Key Responsibilities

What You’ll Bring

• AI & LLM Validation
• Non-Deterministic Testing: Architect automated frameworks to evaluate Generative AI outputs for hallucination, consistency, and factual accuracy against "Gold Standard" datasets.
• RAG Evaluation: Implement automated metrics (e.g., RAGAS, faithfulness, answer relevance) to verify that Retrieval-Augmented Generation pipelines accurately cite technical and regulatory documentation.
• Prompt Regression: Design regression suites to monitor "prompt drift," ensuring model updates do not degrade the quality of AI-generated engineering documents.
• Integration & System Verification
• Enterprise Integration: Build robust tests to validate data consistency between AI agents and critical systems (e.g., SAP S/4HANA, Ariba), ensuring the integrity of Bill of Materials (BOM) and financial data.
• Performance Benchmarking: Design tests to validate latency and throughput for forecasting models and risk-scoring engines using tools like Locust, JMeter, or K6.
• API & Security Validation: Automate testing of secure API gateways, verifying Role-Based Access Control (RBAC) and PII redaction logic before data reaches AI models.
• Governance & Traceability
• V-Model Alignment: Map automated test cases to "System Requirements" to create digital evidence for formal Verification and Validation (V&V) reports.
• Stage Gate Compliance: Prepare "Test Readiness" packages for formal reviews, providing quantitative evidence that systems are stable enough to move from MVP to Production.
• Defect Lifecycle Management: Manage the feedback loop between Requirements Quality Assistants and development teams, tracing AI logic defects back to specific model versions.

Technical Requirements

• Core Automation: Expert proficiency in Python (Pytest) and standard libraries (Selenium/Playwright, Requests).
• AI Evaluation: Hands‑on experience with LLM evaluation frameworks (e.g., DeepEval, TruLens) and "Ground Truth" dataset management.
• Performance Engineering: Proficiency in crafting Performance Test Plans and implementations (Locust, K6, etc.).
• Data Validation: Expertise in SQL and data quality tools (e.g., Great Expectations) for Data Lakehouses and Vector Databases.
• CI/CD & DevOps: Strong experience integrating quality gates into GitLab CI/CD pipelines.
• Engineering Practices: Deep understanding of modern QE practices, including Shift Left, Test Pyramid, and Mono-repo architectures.

Professional Qualifications

• Experience: 5+ years in QA Automation, with 2+ years focused on complex data-driven applications, ML models, or AI agents.
• Domain Expertise: Background in Defense, Aerospace, or highly regulated industries is a strong plus. Familiarity with IV&V processes is highly desirable.
• Analytical Mindset: Ability to define pass/fail criteria for probabilistic systems and communicate "Confidence Levels" to engineering leadership.

Why AI71?

• Final Line of Defense: You define whether an AI agent is "trusted" to negotiate a contract or design a critical system component in a safety-critical environment.
• Tax-Free Compensation: A market-leading, 100% tax-free salary and benefits package.
• Full Relocation Support: Comprehensive assistance including flights, housing support, and visa sponsorship for you and your family to Abu Dhabi.
• Pioneering Work: Set the global standard for how defense organizations validate intelligent systems using world-leading models.

Requirements: