Senior QA Engineer - AI Products (6 Months Contract)

Arabic.AI; Doha; Vor Ort; QAR 180.000 - 240.000; Vollzeit

Zusammenfassung

Arabic.AI is seeking a Senior QA Engineer to own quality across our AI products, testing LLM-powered features end to end and building automated test and evaluation suites. You will define test strategies for web, API, and backend, evaluate model outputs for accuracy and safety, and drive CI/CD integrated automated UI/API tests.

Collaborating with ML engineers and product managers, you will set acceptance criteria, report defects clearly, and help ensure every release is reliable and safe.

Qualifikationen

• Proven experience testing AI/ML or LLM-based products (chatbots, agents, etc.).
• Hands-on scripting in Python and/or TypeScript/JavaScript.
• Hands-on with UI automation frameworks (Playwright, Cypress, Selenium) and a test framework such as Pytest.
• API testing with Postman, REST Assured, or similar.
• CI/CD and version control: GitHub Actions, GitLab CI or Jenkins; Git.
• Test management: Jira, TestRail, or Zephyr.
• Excellent written English and clear bug reporting.

Aufgaben

• Define test strategies and plans for AI features across web, API, and backend.
• Evaluate model outputs for accuracy, consistency, bias, and safety.
• Build evaluation datasets, including edge cases and adversarial prompts.
• Develop and maintain automated UI, API, and regression tests integrated into CI/CD.
• Run regression testing on model, prompt, and configuration changes.
• Test performance, latency, error handling, and fallback behavior.
• Report defects clearly and provide release-readiness sign-off.
• Test non-deterministic outputs using scoring rubrics and human evaluation.

Kenntnisse

QA automation

Python scripting

TypeScript/JavaScript

Tools

Playwright

Cypress

Selenium

Pytest

Postman

REST Assured

GitHub Actions

GitLab CI

Jenkins

Git

Jira

TestRail

Zephyr

Jobbeschreibung

We are looking for a Senior QA Engineer to own quality across our AI products. You will test LLM-powered features end to end, build automated test and evaluation suites, and help ensure every release is accurate, safe, and reliable.

Responsibilities

• Define test strategies and plans for AI features across web, API, and backend.
• Evaluate model outputs for accuracy, consistency, hallucinations, bias, and safety.
• Build evaluation datasets, including edge cases and adversarial / red-team prompts.
• Develop and maintain automated UI, API, and regression tests integrated into CI/CD.
• Run regression testing on model, prompt, and configuration changes.
• Test performance, latency, error handling, and fallback behavior.
• Report defects clearly and provide release-readiness sign-off.
• Test non-deterministic outputs using scoring rubrics, LLM-as-judge, human evaluation, and defined pass/fail thresholds.
• Perform AI security testing, including prompt injection, jailbreaks, and PII / data leakage across chatbots, RAG, and agents.
• Work with ML engineers and product managers to define quality metrics and acceptance criteria.
• Where applicable, monitor production quality and model drift after release.

7+ years of experience in QA / software testing, with strong test automation background.

Requirements

• Proven experience testing AI/ML or LLM-based products (chatbots, RAG, agents, etc.).
• Hands-on scripting experience in Python and/or TypeScript/JavaScript.
• Hands-on with at least one UI automation framework (Playwright, Cypress, Selenium) and a test framework such as Pytest.
• API testing with Postman, REST Assured, or similar.
• Experience with LLM evaluation tools.
• CI/CD and version control: GitHub Actions, GitLab CI or Jenkins; Git.
• Test management and tracking: Jira, TestRail, or Zephyr.
• Excellent written English and clear bug reporting.

Nice to Have

• Performance/load testing (k6, JMeter, Locust).
• Understanding of prompt engineering, embeddings, and vector databases.
• Experience with AI safety, red-teaming, or content policy testing.
• Cloud experience (AWS, GCP, or Azure) and Docker.