AI - Software Development Engineer in Test (AI - SDET)

Web.com Canada, Inc.; Argentina; On-site; ARS 1,200,000 - 2,400,000; Full time

Job summary

Newfold Digital is seeking an AI Quality Engineer to validate, evaluate, and improve AI-powered applications, copilots, and LLM-driven workflows. You will build scalable automation frameworks, ensure reliability, safety, observability, and quality across AI-native systems, agents, and multi-service environments.

You will work with Engineering, AI/ML, Platform, Product, and DevOps teams to implement testing strategies, evaluation frameworks, and CI/CD integrations that drive quality and

Qualifications

• Strong programming experience in Python and solid software testing fundamentals.
• Hands-on experience with AI/LLM testing, prompt engineering, and context engineering.
• Experience with CI/CD pipelines, Docker & Kubernetes, and cloud platforms (AWS/Azure/OCI/GCP).
• API testing & automation, microservices testing, and UI automation.

Responsibilities

• Design, validate, and optimise testing strategies for AI-powered applications and autonomous agents.
• Develop AI evaluation frameworks for hallucination, groundedness, toxicity bias, and safety validation.
• Build automated validation pipelines for LLM outputs using semantic scoring and evaluation metrics.
• Create synthetic and benchmark datasets for AI testing scenarios.
• Test multi-agent orchestration workflows and MCP integrations.
• Validate AI tool interoperability and context/prompt optimization strategies.
• Monitor latency, token usage, and scalability in AI-native systems.

Skills

Python

QA principles

AI/LLM testing

CI/CD pipelines

Docker & Kubernetes

Cloud platforms

SQL & data validation

Agile/Scrum

UI automation

API testing & automation

Event-driven testing

Tools

Robot Framework

Playwright

LangChain

LangGraph

OpenAI APIs

Weaviate

Pinecone

FAISS

Job description

This role goes beyond traditional QA; the ideal candidate should be capable of validating, evaluating, and improving AI-powered systems, agents, copilots, and LLM-driven workflows while building scalable automation frameworks and intelligent quality solutions. You will work closely with Engineering, AI/ML, Platform, Product, and DevOps teams to ensure reliability, safety, observability, scalability, and quality of modern AI-native applications and Agentic systems.

Who Are We.

Newfold Digital is a leading web technology company serving nearly seven million customers globally. Established in 2021 through the combination of leading web services providers Endurance Web Presence and Web.com Group, our portfolio of brands includes: Bluehost, BigRock, ResellerClub, CrazyDomains, HostGator, Network Solutions, Register.com, Web.com and many others.

We help customers of all sizes build a digital presence that delivers results. With our extensive product offerings and personalized support, we take pride in collaborating with our customers to serve their online presence needs.

Key Responsibilities

AI/LLM Quality Engineering

• Design, validate, and optimise testing strategies for AI-powered applications, copilots, chatbots, RAG pipelines, and autonomous AI agents.
• Develop AI evaluation (Evals) frameworks for:Hallucination detectionGroundednessToxicity & bias validationResponse qualityContext relevancePrompt regression testingDeterminism & consistencySafety and guardrail validation
• Build automated validation pipelines for LLM outputs using structured assertions, semantic scoring, embeddings, and evaluation metrics.
• Create synthetic datasets and benchmark datasets for AI testing scenarios.
• Test multi-agent orchestration workflows and agent memory/context retention.
• Validate MCP (Model Context Protocol) integrations and tool interoperability.
• Work on Context Engineering, Prompt Engineering, and Prompt Optimisation strategies.
• Evaluate latency, token usage, model cost optimisation, throughput, and scalability.

Automation Engineering

• Develop and maintain scalable UI/API automation frameworks using:Python (mandatory)Robot Framework (preferred)Playwright/Selenium/Cypress
• Automate testing for:REST APIsGraphQL APIsEvent-driven systemsStreaming systemsWebhooksMicroservicesMFE (Micro Frontend) architectures
• Build reusable automation libraries, utilities, and AI-assisted testing workflows.
• Integrate automation into CI/CD pipelines and quality gates.

AI Engineering & Platform Validation

• Validate RAG architectures including:Chunking strategiesEmbedding qualityRetrieval accuracyRe-rankingVector search relevance
• Work with vector databases such as:PineconeWeaviateChromaDBQdrantFAISS
• Build or validate AI agents leveraging:LangChainLangGraphCrewAIAutoGenSemantic KernelOpenAI SDKs
• Validate AI workflows involving:Tool callingFunction callingAgent memoryAgent planning/executionMulti-modal AI systemsStructured outputs

DevOps, Observability & Reliability

• Build and maintain quality workflows integrated with:GitHub ActionsJenkinsGitLab CI/CDAzure DevOps
• Work with Docker and Kubernetes environments.
• Perform cloud-based testing and deployments across AWS/Azure/OCI/GCP.
• Validate distributed systems reliability and resiliency.
• Monitor AI system behaviour using observability platforms/logging/tracing tools.
• Contribute to shift-left and quality engineering best practices.

Required Skills

Must Have

• Strong programming experience in Python
• Strong understanding of Software Testing & Quality Engineering principles
• Hands-on experience with:AI/LLM testingPrompt EngineeringContext EngineeringAI EvalsRAG systemsAI Agents
• Experience with:CI/CD pipelinesDocker & KubernetesCloud platforms (AWS/Azure/OCI/GCP)API testing & automationMicroservices testingUI automation
• Experience with SQL and data validation
• Strong debugging, analytical, and troubleshooting skills
• Experience working in Agile/Scrum environments

Preferred / Good to Have

• Experience with Robot Framework
• Experience with Playwright
• Knowledge of:LangChainLangGraphLlamaIndexCrewAISemantic KernelMCP (Model Context Protocol)
• Hands‑on experience with:CursorGitHub CopilotClaudeGeminiOpenAI APIs
• Experience with vector databases and embeddings
• Exposure to MLOps workflows and ML lifecycle testing
• Knowledge of performance testing and security testing
• Experience validating AI governance/compliance requirements
• Experience with contract testing, chaos testing, or resilience engineering
• Familiarity with accessibility testing and cross-browser/device testing
• Exposure to synthetic monitoring and production observability

What We're Looking For

• Engineers who actively experiment with AI tools and build side projects, prototypes, PoCs, or open-source contributions.
• Strong problem solvers who can combine traditional QE practices with modern AI-native engineering approaches.
• Passion for innovation, developer productivity, automation acceleration, and intelligent testing.
• Ability to rapidly learn emerging AI frameworks, tools, a