Software Development Engineer in Test - AI

Web.com Canada, Inc.; India; On-site; INR 11,353,000 - 17,029,000; Full time

Job summary

Newfold Digital is seeking an AI/LLM Quality Engineer who will validate, evaluate, and improve AI-powered systems, agents, and workflows while building scalable automation frameworks. You will collaborate with Engineering, AI/ML, Platform, Product, and DevOps teams to ensure reliability, safety, observability, and scalability of modern AI-native applications.

You will design testing strategies, build automated validation pipelines, develop Evals, and help shape context and prompt optimization.

Qualifications

• Bachelor's or Master's degree in CS/Engineering or related field.
• Hands-on experience with Python and AI/LLM testing.
• Strong understanding of QA principles and automation.

Responsibilities

• Design, validate, and optimize testing strategies for AI-powered apps, copilots, chatbots, RAG pipelines, and autonomous AI agents.
• Develop AI evaluation (Evals) frameworks for hallucination, groundedness, toxicity, bias, and response quality.
• Build automated validation pipelines for LLM outputs using semantic scoring and evaluation metrics.

Skills

Python

CI/CD

QA engineering

AI/LLM testing

Debugging

Agile/Scrum

Education

Bachelor's/Master's in CS or related

Tools

Robot Framework

Playwright

Selenium

Cypress

Jenkins

GitHub Actions

Docker

Kubernetes

LangChain

OpenAI SDKs

Job description

Who we are.

Newfold Digital is a leading web technology company serving millions of customers globally. Our customers know us through our robust portfolio of brands. We have some of the industry's most prominent and storied go-to-market brands, including Bluehost, HostGator, Domain.com, Network Solutions, Register.com and Web.com. We help customers of all sizes build a digital presence that delivers results. With our extensive product offerings and personalized support, we take pride in collaborating with our customers to serve their online presence needs. The strength of our company lives in the intersection of our people, our customers, and our brands.

Role Summary:

This role goes beyond traditional QA; the ideal candidate should be capable of validating, evaluating, and improving AI-powered systems, agents, copilots, and LLM-driven workflows while building scalable automation frameworks and intelligent quality solutions. You will work closely with Engineering, AI/ML, Platform, Product, and DevOps teams to ensure reliability, safety, observability, scalability, and quality of modern AI-native applications and Agentic systems.

What you'll do & how you'll make your mark.

AI/LLM Quality Engineering

• Design, validate, and optimize testing strategies for AI-powered applications, copilots, chatbots, RAG pipelines, and autonomous AI agents.
• Develop AI evaluation (Evals) frameworks for: Hallucination detection Groundedness Toxicity & bias validation Response quality Context relevance Prompt regression testing Determinism & consistency Safety and guardrail validation
• Build automated validation pipelines for LLM outputs using structured assertions, semantic scoring, embeddings, and evaluation metrics.
• Create synthetic datasets and benchmark datasets for AI testing scenarios.
• Test multi-agent orchestration workflows and agent memory/context retention.
• Validate MCP (Model Context Protocol) integrations and tool interoperability.
• Work on Context Engineering, Prompt Engineering, and Prompt Optimization strategies.
• Evaluate latency, token usage, model cost optimization, throughput, and scalability.

Automation Engineering

• Develop and maintain scalable UI/API automation frameworks using: Python (mandatory)Robot Framework (preferred)Playwright/Selenium/Cypress.
• Automate testing for: REST APIs GraphQL APIs Event-driven systems Streaming systems Webhooks Microservices MFE (Micro Frontend) architectures.
• Build reusable automation libraries, utilities, and AI-assisted testing workflows.
• Integrate automation into CI/CD pipelines and quality gates.

AI Engineering & Platform Validation

• Validate RAG architectures including Chunking strategies Embedding quality Retrieval accuracy Re-ranking Vector search relevance
• Work with vector databases such as: Pinecone Weaviate ChromaDB Qdrant FAISS
• Build or validate AI agents leveraging: LangChain, LangGraph, CrewAI, AutoGen, Semantic Kernel OpenAI SDKs
• Validate AI workflows involving: Tool calling Function calling Agent memory Agent planning/execution Multi-modal AI systems Structured outputs

DevOps, Observability & Reliability

• Build and maintain quality workflows integrated with: GitHub Actions Jenkins GitLab CI/CD Azure DevOps
• Work with Docker and Kubernetes environments.
• Perform cloud-based testing and deployments across AWS/Azure/OCI/GCP.
• Validate distributed systems reliability and resiliency.
• Monitor AI system behaviors using observability platforms/logging/tracing tools.
• Contribute to shift-left and quality engineering best practices.

Who you are & what you'll need to succeed.

• Bachelor's/Master's degree in Computer Science, Engineering, AI/ML, or related field preferred.
• Equivalent hands-on experience and demonstrated projects are highly valued.
• Strong programming experience in Python
• Strong understanding of Software Testing & Quality Engineering principles
• Hands-on experience with: AI/LLM testing Prompt Engineering Context Engineering AI Evals RAG systems AI Agents
• Experience with: CI/CD pipelines Docker & KubernetesCloud platforms (AWS/Azure/OCI/GCP) API testing & automation Microservices testing UI automation
• Experience with SQL and data validation
• Strong debugging, analytical, and troubleshooting skills
• Experience working in Agile/Scrum environments

Good to have skills:

• Experience with Robot Framework
• Experience with Playwright
• Knowledge of: LangChain, LangGraph, LlamaIndex CrewAI Semantic Kernel MCP (Model Context Protocol)
• Hands-on experience with: Cursor GitHub Copilot Claude Gemini OpenAI APIs
• Experience with vector databases and embeddings
• Exposure to MLOps workflows and ML lifecycle testing
• Knowledge of performance testing and security testing
• Experience validating AI governance/compliance requirements
• Experience with contract testing, chaos testing, or resilience engineering
• Familiarity with accessibility testing and cross-browser/device testing
• Exposure to synthetic monitoring and production observability

Bonus Skills (High Impact)

• AI Red Teaming
• LLM Se