AI Quality Engineer: LLM & Agentic Systems Testing

Web.com Canada, Inc.

Argentina

On-site

ARS 1,200,000 - 2,400,000

Full time

9 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Newfold Digital is seeking an AI Quality Engineer to validate, evaluate, and improve AI-powered applications, copilots, and LLM-driven workflows. You will build scalable automation frameworks, ensure reliability, safety, observability, and quality across AI-native systems, agents, and multi-service environments.

You will work with Engineering, AI/ML, Platform, Product, and DevOps teams to implement testing strategies, evaluation frameworks, and CI/CD integrations that drive quality and

Qualifications

  • Strong programming experience in Python and solid software testing fundamentals.
  • Hands-on experience with AI/LLM testing, prompt engineering, and context engineering.
  • Experience with CI/CD pipelines, Docker & Kubernetes, and cloud platforms (AWS/Azure/OCI/GCP).
  • API testing & automation, microservices testing, and UI automation.

Responsibilities

  • Design, validate, and optimise testing strategies for AI-powered applications and autonomous agents.
  • Develop AI evaluation frameworks for hallucination, groundedness, toxicity bias, and safety validation.
  • Build automated validation pipelines for LLM outputs using semantic scoring and evaluation metrics.
  • Create synthetic and benchmark datasets for AI testing scenarios.
  • Test multi-agent orchestration workflows and MCP integrations.
  • Validate AI tool interoperability and context/prompt optimization strategies.
  • Monitor latency, token usage, and scalability in AI-native systems.

Skills

Python
QA principles
AI/LLM testing
CI/CD pipelines
Docker & Kubernetes
Cloud platforms
SQL & data validation
Agile/Scrum
UI automation
API testing & automation
Event-driven testing

Tools

Robot Framework
Playwright
LangChain
LangGraph
OpenAI APIs
Weaviate
Pinecone
FAISS

Job description

Newfold Digital is seeking an AI Quality Engineer to validate, evaluate, and improve AI-powered applications, copilots, and LLM-driven workflows. You will build scalable automation frameworks, ensure reliability, safety, observability, and quality across AI-native systems, agents, and multi-service environments.

You will work with Engineering, AI/ML, Platform, Product, and DevOps teams to implement testing strategies, evaluation frameworks, and CI/CD integrations that drive quality and

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI - Software Development Engineer in Test (AI - SDET)
AI - Software Development Engineer in Test (AI - SDET)

Web.com Canada, Inc. • Argentina

On-site
ARS 1,200,000 - 2,400,000
AI Quality Engineer
AI Quality Engineer

Moveo Technologies Corp - drvn • Rosario

On-site
ARS 1,200,000 - 2,400,000
AI Quality Engineer
AI Quality Engineer

drvn • Rosario

On-site
ARS 1,800,000 - 3,000,000
AI-Driven QA Automation Engineer
AI-Driven QA Automation Engineer

Newfold Digital • Argentina

Hybrid
ARS 130,771,000 - 184,618,000
OSDE 310 Health Care
Annual Bonus
Professional Development
Remote QA Automation Engineer for AI/ML & Quality Engineering
Remote QA Automation Engineer for AI/ML & Quality Engineering

Strategic Staffing Solutions • Quilmes

Hybrid
ARS 92,724,000 - 139,086,000
Professional growth
Competitive USD-based compensation
Exciting projects with Fortune 500/top
+1
AI QA Automation Engineer
AI QA Automation Engineer

Newfold Digital • Argentina

Hybrid
ARS 130,771,000 - 184,618,000
OSDE 310 Health Care
Annual Bonus
Professional Development
AI-Driven QA Architect for Intelligent Testing
AI-Driven QA Architect for Intelligent Testing

Moveo Technologies Corp - drvn • Rosario

On-site
ARS 1,200,000 - 2,400,000
Senior AI/LLM Automation QA Engineer
Senior AI/LLM Automation QA Engineer

Eleks • Buenos Aires

On-site
ARS 138,487,000 - 200,037,000
Close collaboration with a customer
Challenging tasks
Competence development
+3
AI Quality Engineer: Automation & Observability Lead
AI Quality Engineer: Automation & Observability Lead

drvn • Rosario

On-site
ARS 1,800,000 - 3,000,000
Senior Automation Test Engineer – AI/LLM, PyTest & CI/CD
Senior Automation Test Engineer – AI/LLM, PyTest & CI/CD

Eleks • Argentina

On-site
ARS 1,200,000 - 2,100,000