AI Evaluation & Quality Assurance Engineer

Datamatics Careers- Philippines; Pasig; On-site; PHP 1,200,000 - 2,100,000; Full time

Job summary

Datamatics Careers- Philippines is seeking a QA & AI Evaluation Engineer to establish robust test and evaluation strategies for AI-enabled workflows, including LLM-powered chat and automation systems. You will design test suites, evaluate models, and communicate findings to engineering and operations teams.

The role emphasizes rigorous experimentation, data privacy, and handling complex test datasets with business SMEs, driving release readiness and risk mitigation.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, Data Science, Statistics or a related discipline, or equivalent practical experience.
• Experience in software QA, test automation, data/AI evaluation, applied research or quality engineering; exposure to LLM-enabled products is strongly relevant.
• Practical programming ability, preferably in Python, for test automation, data analysis, evaluation pipelines or API-based experiments.
• Understanding of LLM and RAG systems, conversational/agent workflows and quality concerns such as grounding, hallucination, prompt injection and model variability.
• Familiarity with experimental design, metrics, test datasets, defect lifecycle and communicating evidence-based findings to technical and business audiences.

Responsibilities

• Define and maintain test and evaluation strategies for AI and automation use cases, covering task success, correctness, groundedness, safety, consistency, latency and cost.
• Create representative test sets, reference answers, rubrics, edge cases and regression suites with business SMEs; protect sensitive data and document dataset limitations.
• Evaluate LLM, RAG, agent and conversational workflows using appropriate automated and human review methods; analyze failure patterns and communicate evidence clearly.
• Design functional, integration, end-to-end, load and regression tests for conversational and AI-enabled solutions, including tool calls, handoffs, fallback behavior and exception paths.
• Conduct adversarial and safety testing for hallucinations, prompt injection, unsafe or biased outputs, data leakage, inappropriate actions and control bypasses; coordinate remediation with owners.
• Benchmark approved models, prompts, retrieval configurations and platforms against agreed use-case criteria; run controlled experiments and document trade-offs and reproducible results.
• Research relevant advances in LLM evaluation, retrieval, agents, responsible AI and testing methods; translate useful findings into practical prototypes or recommendations.
• Build evaluation scripts, dashboards and reporting that make quality trends, release readiness, known limitations and post-deployment drift visible to delivery and governance stakeholders.
• Partner with AI/LLM Engineering, Solution Architecture, Software/Integration and Operations to define acceptance thresholds, triage defects and verify fixes before release.

Skills

Python

LLM & RAG systems

experimental design

data analysis

QA & test automation

Education

Bachelor’s degree in Computer Science or related

Tools

Python

Job description

Datamatics Careers- Philippines is seeking a QA & AI Evaluation Engineer to establish robust test and evaluation strategies for AI-enabled workflows, including LLM-powered chat and automation systems. You will design test suites, evaluate models, and communicate findings to engineering and operations teams.

The role emphasizes rigorous experimentation, data privacy, and handling complex test datasets with business SMEs, driving release readiness and risk mitigation.