AI QA & Evaluation Engineer

Boundless; Dubai; Sur place; AED 260 000 - 420 000; Plein temps

Résumé du poste

Boundless is seeking an experienced QA/Quality Engineer to establish and run evaluation practices for AI systems.

You will design risk-based tests for generative AI, validate end-to-end workflows, and develop robust rubrics and datasets with business experts.

Qualifications

• 5+ years in software QA, test automation or quality engineering, incl. AI/ML or data-driven systems
• Strong Python, API testing, SQL and automated test framework skills, with CI/CD experience
• Understanding of LLM variability, retrieval systems and limitations of exact-match testing
• Ability to design representative test data, evaluation rubrics and thresholds with stakeholders
• Knowledge of statistical metrics, sampling and data separation in training/validation/test sets
• Experience testing access controls, sensitive data handling and production recoverability
• Bachelor’s degree in CS/Engineering or related field, or equivalent experience

Responsabilités

• Define risk-based test strategies for generative AI, agent workflows, predictive models and automation integrations
• Create representative, versioned evaluation datasets with business experts
• Measure grounded accuracy, retrieval relevance, citation correctness and end-to-end task completion
• Develop scoring rubrics and reference answers and calibrate judges
• Test prompt injection, data leakage, cross-client access and excessive permissions
• Validate end-to-end outcomes across connected systems and ensure proper handoff to humans
• Integrate regression evaluations into CI/CD and maintain baselines and thresholds
• Reproduce key results with governance and maintain test records

Connaissances

Python

API testing

SQL

Test automation

LLM evaluation

CI/CD

Formation

Bachelor's degree in Computer Science or Engineering

Outils

Postman

PyTest

Selenium

JMeter

Description du poste

Establish and run the quality assurance and evaluation practices needed to release AI systems safely and reliably. Test complete workflows as well as individual models, covering accuracy, retrieval quality, agent actions, security, robustness and operational behaviour. Provide clear, repeatable evidence of whether each AI release meets its intended use and agreed acceptance criteria.

Key responsibilities

Evaluation design and test data

• Define risk-based test strategies for generative AI, agent workflows, predictive models and automation integrations, matching coverage to intended use and potential impact.
• Create representative, versioned evaluation datasets with business experts, including normal cases, ambiguous requests, rare exceptions and sensitive brokerage scenarios.
• Measure grounded accuracy, retrieval relevance, citation correctness, structured-output validity, task completion and appropriate escalation to a human.
• Develop scoring rubrics and reviewed reference answers; calibrate automated or LLM-based judges against human assessments rather than relying on a single model score.

Adversarial and end-to-end testing

• Test prompt injection, indirect malicious instructions, data leakage, cross-client access, excessive agent permissions and attempts to bypass approval steps.
• Validate end-to-end outcomes across CRM, support and other connected systems, including duplicate actions, partial completion, timeouts, retries and recovery.
• Test brokerage-specific risks such as inaccurate product terms, outdated policies, unsupported trading advice, inappropriate client messaging and incorrect account information.
• Evaluate latency, concurrency, provider outages, degraded retrieval and model changes; confirm that fallback and human handoff behave as designed.

Release assurance and continuous evaluation

• Integrate regression evaluations into CI/CD and compare candidate releases against approved baselines with consistent datasets and documented thresholds.
• Independently reproduce key quantitative test results with the Quantitative Lead, checking dataset separation, leakage controls and stability across relevant segments.
• Maintain defect triage, release evidence and exception records; challenge release readiness and escalated unresolved failures through the agreed governance process.
• Use redacted production samples, incidents and user feedback to extend test coverage while keeping development data separate from held‑out evaluation sets.

Essential experience and qualifications

• 5+ years in software QA, test automation or quality engineering, including hands‑on evaluation of AI, ML or data‑driven systems.
• Strong Python, API testing, SQL and automated test framework skills, with experience integrating tests into delivery pipelines.
• Understanding of LLM variability, retrieval systems, agent tool execution and the limitations of exact‑match testing for generated outputs.
• Ability to design representative test data, evaluation rubrics and meaningful thresholds with business and technical stakeholders.
• Working knowledge of statistical metrics, sampling, class imbalance and separation of training, validation and test data.
• Experience testing access controls, sensitive data handling, integration failures and recoverability in production‑oriented systems.
• Bachelor’s degree in Computer Science, Engineering, Mathematics or a related field, or equivalent relevant experience.

Preferred experience

• Financial services QA experience, especially brokerage platforms, payment processes or regulated client communications.
• Experience with AI red teaming, human annotation and multilingual evaluations.
• Experience assessing predictive models, drift monitoring and model documentation.