LLM QA Automation Engineer — AI Model Evaluation

Tektora; Abu Dhabi; Presencial; AED 300.000 - 480.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Full relocation support

Visa sponsorship

Family status available

Descripción de la vacante

Tektora are seeking QA AI Automation Engineers to join an on-site, permanent role in Abu Dhabi. You will own evaluation of internal AI models, building Python-based test frameworks, ensuring output quality and coverage across production and synthetic data.

You will design evaluation strategies, create golden datasets, and integrate tests into CI/CD. Relocation and visa sponsorship are provided for a full move to Abu Dhabi.

Formación

• Bachelor’s or Master’s degree.
• Strong Python — scripting, testing frameworks such as pytest, and working with APIs.
• Experience testing AI or LLM outputs, or comparable ML system validation, using evaluation frameworks such as RAGAS, DeepEval or LangSmith, or custom scoring of your own
• Practical data handling — cleaning real data, generating and synthesising test data, and building golden datasets
• Familiarity with CI/CD pipelines such as GitLab CI or GitHub Actions
• Basic SQL and NoSQL alongside working data analysis
• Experience with FastAPI, Docker and Linux

Responsabilidades

• Building and maintaining Python automation test suites and frameworks from scratch that continuously validate the company’s internal AI models
• Testing LLM outputs for quality, accuracy, consistency, edge cases and failure modes
• Building and managing test datasets that combine real production data with synthetic data — generated scenarios, edge cases and adversarial inputs — to maximise coverage
• Writing automated test scripts that run on every model and prompt update
• Running regression tests, golden-dataset evaluations and performance benchmarks across latency, cost and throughput
• Integrating evaluation into CI/CD with quality gates that block deployment when thresholds drop
• Tracking and reporting model performance over time through dashboards and alerting so regressions are caught early
• Working with the AI, ML and product teams to define what good output looks like per use case and encoding that into automated checks

Conocimientos

Python

APIs

Testing frameworks (pytest)

AI/LLM evaluation

CI/CD practices

SQL/NoSQL

FastAPI

Educación

Bachelor’s or Master’s degree

Herramientas

Docker

Linux

GitLab CI

GitHub Actions

Descripción del empleo

Tektora are seeking QA AI Automation Engineers to join an on-site, permanent role in Abu Dhabi. You will own evaluation of internal AI models, building Python-based test frameworks, ensuring output quality and coverage across production and synthetic data.

You will design evaluation strategies, create golden datasets, and integrate tests into CI/CD. Relocation and visa sponsorship are provided for a full move to Abu Dhabi.