QA AI Automation Engineer - LLM And Model Evaluation

Tektora; Abu Dhabi; On-site; AED 300,000 - 480,000; Full time

Benefits offered by this job

Full relocation support

Visa sponsorship

Family status available

Job summary

Tektora are seeking QA AI Automation Engineers to join an on-site, permanent role in Abu Dhabi. You will own evaluation of internal AI models, building Python-based test frameworks, ensuring output quality and coverage across production and synthetic data.

You will design evaluation strategies, create golden datasets, and integrate tests into CI/CD. Relocation and visa sponsorship are provided for a full move to Abu Dhabi.

Qualifications

• Bachelor’s or Master’s degree.
• Strong Python — scripting, testing frameworks such as pytest, and working with APIs.
• Experience testing AI or LLM outputs, or comparable ML system validation, using evaluation frameworks such as RAGAS, DeepEval or LangSmith, or custom scoring of your own
• Practical data handling — cleaning real data, generating and synthesising test data, and building golden datasets
• Familiarity with CI/CD pipelines such as GitLab CI or GitHub Actions
• Basic SQL and NoSQL alongside working data analysis
• Experience with FastAPI, Docker and Linux

Responsibilities

• Building and maintaining Python automation test suites and frameworks from scratch that continuously validate the company’s internal AI models
• Testing LLM outputs for quality, accuracy, consistency, edge cases and failure modes
• Building and managing test datasets that combine real production data with synthetic data — generated scenarios, edge cases and adversarial inputs — to maximise coverage
• Writing automated test scripts that run on every model and prompt update
• Running regression tests, golden-dataset evaluations and performance benchmarks across latency, cost and throughput
• Integrating evaluation into CI/CD with quality gates that block deployment when thresholds drop
• Tracking and reporting model performance over time through dashboards and alerting so regressions are caught early
• Working with the AI, ML and product teams to define what good output looks like per use case and encoding that into automated checks

Skills

Python

APIs

Testing frameworks (pytest)

AI/LLM evaluation

CI/CD practices

SQL/NoSQL

FastAPI

Education

Bachelor’s or Master’s degree

Tools

Docker

Linux

GitLab CI

GitHub Actions

Job description

Job Description

Location: Abu Dhabi, United Arab Emirates

Working Pattern: On-site, 5 days per week

Type: Permanent, full-time

Salary: Competitive tax-free package, dependent on experience

Relocation: Full relocation support, visa sponsorship and family status available

Company: Established technology developer (name shared at first conversation)

Tektora are supporting an established technology client in Abu Dhabi who are looking to speak with QA AI Automation Engineers for a permanent opportunity owning the testing strategy for their internal AI and LLM models.

This is not a conventional QA role, and it is not a role about using AI to test software. The thing under test here is the AI itself. You will own how the company proves its own models behave — output quality, accuracy, consistency, edge cases and failure modes — across a mix of real production data and synthetic test data, and you will build the Python frameworks that do it from scratch.

Relocation to Abu Dhabi is required. This is an on-site environment and the role cannot be performed remotely. The package is tax-free, relocation is fully supported, and the client has moved engineers and their families into this programme before.

This role would suit

A Python-strong engineer who has moved into AI and LLM evaluation — someone who has built eval harnesses rather than only consumed them, who has worked with golden datasets, regression suites and quality gates on model and prompt changes, and who is comfortable arguing about what “good output” actually means for a given use case. You might currently call yourself a QA automation engineer, an ML or MLOps engineer, an AI evaluation engineer or a data engineer; the title matters far less than having genuinely tested model behaviour in production.

Background can be commercial software, AI product, data platform or research engineering. Defence or national-security experience is welcome but is not required for this role.

You Will Be Involved In

• Building and maintaining Python automation test suites and frameworks from scratch that continuously validate the company’s internal AI models
• Testing LLM outputs for quality, accuracy, consistency, edge cases and failure modes
• Building and managing test datasets that combine real production data with synthetic data — generated scenarios, edge cases and adversarial inputs — to maximise coverage
• Writing automated test scripts that run on every model and prompt update
• Running regression tests, golden-dataset evaluations and performance benchmarks across latency, cost and throughput
• Integrating evaluation into CI/CD with quality gates that block deployment when thresholds drop
• Tracking and reporting model performance over time through dashboards and alerting so regressions are caught early
• Working with the AI, ML and product teams to define what good output looks like per use case and encoding that into automated checks

Key Experience Required

• Bachelor’s or Master’s degree
• Strong Python — scripting, testing frameworks such as pytest, and working with APIs
• Experience testing AI or LLM outputs, or comparable ML system validation, using evaluation frameworks such as RAGAS, DeepEval or LangSmith, or custom scoring of your own
• Practical data handling — cleaning real data, generating and synthesising test data, and building golden datasets
• Familiarity with CI/CD pipelines such as GitLab CI or GitHub Actions
• Basic SQL and NoSQL alongside working data analysis
• Experience with FastAPI, Docker and Linux

Desirable Experience

• Working knowledge of vector search and embeddings
• Model fine-tuning concepts, prompt engineering or observability tooling
• MCP (Model Context Protocol)
• GPU and CPU optimisation
• ISTQB CT-AI certification

Interested?

Tektora are looking to speak with QA AI Automation Engineers and AI evaluation engineers who are open to a permanent move to Abu Dhabi and keen to own model quality end to end.