QA / AI Evaluation & Research Engineer

Datamatics Careers- Philippines; Pasig; Presencial; PHP 1.200.000 - 2.100.000; Jornada completa

Descripción de la vacante

Datamatics Careers- Philippines is seeking a QA & AI Evaluation Engineer to establish robust test and evaluation strategies for AI-enabled workflows, including LLM-powered chat and automation systems. You will design test suites, evaluate models, and communicate findings to engineering and operations teams.

The role emphasizes rigorous experimentation, data privacy, and handling complex test datasets with business SMEs, driving release readiness and risk mitigation.

Formación

• Bachelor’s degree in Computer Science, Engineering, Data Science, Statistics or a related discipline, or equivalent practical experience.
• Experience in software QA, test automation, data/AI evaluation, applied research or quality engineering; exposure to LLM-enabled products is strongly relevant.
• Practical programming ability, preferably in Python, for test automation, data analysis, evaluation pipelines or API-based experiments.
• Understanding of LLM and RAG systems, conversational/agent workflows and quality concerns such as grounding, hallucination, prompt injection and model variability.
• Familiarity with experimental design, metrics, test datasets, defect lifecycle and communicating evidence-based findings to technical and business audiences.

Responsabilidades

• Define and maintain test and evaluation strategies for AI and automation use cases, covering task success, correctness, groundedness, safety, consistency, latency and cost.
• Create representative test sets, reference answers, rubrics, edge cases and regression suites with business SMEs; protect sensitive data and document dataset limitations.
• Evaluate LLM, RAG, agent and conversational workflows using appropriate automated and human review methods; analyze failure patterns and communicate evidence clearly.
• Design functional, integration, end-to-end, load and regression tests for conversational and AI-enabled solutions, including tool calls, handoffs, fallback behavior and exception paths.
• Conduct adversarial and safety testing for hallucinations, prompt injection, unsafe or biased outputs, data leakage, inappropriate actions and control bypasses; coordinate remediation with owners.
• Benchmark approved models, prompts, retrieval configurations and platforms against agreed use-case criteria; run controlled experiments and document trade-offs and reproducible results.
• Research relevant advances in LLM evaluation, retrieval, agents, responsible AI and testing methods; translate useful findings into practical prototypes or recommendations.
• Build evaluation scripts, dashboards and reporting that make quality trends, release readiness, known limitations and post-deployment drift visible to delivery and governance stakeholders.
• Partner with AI/LLM Engineering, Solution Architecture, Software/Integration and Operations to define acceptance thresholds, triage defects and verify fixes before release.

Conocimientos

Python

LLM & RAG systems

experimental design

data analysis

QA & test automation

Educación

Bachelor’s degree in Computer Science or related

Herramientas

Python

Descripción del empleo

Establish confidence in AI-enabled solutions through systematic testing, evaluation and applied research. The role develops quality frameworks and evaluation assets for LLM, agent, chatbot and automation workflows, measures performance against business and risk criteria, and turns findings into clear recommendations for engineering and operational teams.

KEY RESPONSIBILITIES

• Define and maintain test and evaluation strategies for AI and automation use cases, covering task success, correctness, groundedness, safety, consistency, latency and cost.
• Create representative test sets, reference answers, rubrics, edge cases and regression suites with business SMEs; protect sensitive data and document dataset limitations.
• Evaluate LLM, RAG, agent and conversational workflows using appropriate automated and human review methods; analyze failure patterns and communicate evidence clearly.
• Design functional, integration, end-to-end, load and regression tests for conversational and AI-enabled solutions, including tool calls, handoffs, fallback behavior and exception paths.
• Conduct adversarial and safety testing for hallucinations, prompt injection, unsafe or biased outputs, data leakage, inappropriate actions and control bypasses; coordinate remediation with owners.
• Benchmark approved models, prompts, retrieval configurations and platforms against agreed use-case criteria; run controlled experiments and document trade-offs and reproducible results.
• Research relevant advances in LLM evaluation, retrieval, agents, responsible AI and testing methods; translate useful findings into practical prototypes or recommendations.
• Build evaluation scripts, dashboards and reporting that make quality trends, release readiness, known limitations and post-deployment drift visible to delivery and governance stakeholders.
• Partner with AI/LLM Engineering, Solution Architecture, Software/Integration and Operations to define acceptance thresholds, triage defects and verify fixes before release.

QUALIFICATIONS & EXPERIENCE

• Bachelor’s degree in Computer Science, Engineering, Data Science, Statistics or a related discipline, or equivalent practical experience.
• Experience in software QA, test automation, data/AI evaluation, applied research or quality engineering; exposure to LLM-enabled products is strongly relevant.
• Practical programming ability, preferably in Python, for test automation, data analysis, evaluation pipelines or API-based experiments.
• Understanding of LLM and RAG systems, conversational/agent workflows and quality concerns such as grounding, hallucination, prompt injection and model variability.
• Familiarity with experimental design, metrics, test datasets, defect lifecycle and communicating evidence-based findings to technical and business audiences.

CORE COMPETENCIES

• Scientific curiosity, rigor, reproducibility and a healthy skepticism of unverified model outputs.
• Analytical debugging, attention to detail and ability to prioritize failures by user impact and risk.
• Clear collaboration across engineering, operations, business SMEs, security and governance.

KEY STAKEHOLDERS

• AI & Transformation Lead; AI/LLM Engineer; Solution Architect/Business Analyst; Software/Integration Developer; Operations and business SMEs; Security, Privacy and Governance; platform vendors.

KEY SUCCESS MEASURES

• Coverage, repeatability and usefulness of test datasets, evaluation frameworks and regression suites.
• Quality and risk issues identified before release, with clear evidence, prioritization and verified remediation.
• Performance of solutions against agreed task, safety, reliability, latency and cost thresholds.
• Timeliness and practical value of research findings, experiments and recommendations adopted by delivery teams.