AI - Software Development Engineer in Test (AI - SDET)

Newfold Digital

Maharashtra

On-site

INR 900,000 - 1,500,000

Full time

8 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Newfold Digital in Maharashtra, India, seeks an AI/LLM Quality Engineer to validate, evaluate, and improve AI-powered systems, agents, copilots, and LLM-driven workflows while building scalable automation frameworks and intelligent quality solutions.

You will collaborate with Engineering, AI/ML, Platform, Product, and DevOps teams to ensure reliability, safety, observability, scalability, and quality of modern AI-native applications and Agentic systems.

Qualifications

  • Strong programming experience in Python.
  • Hands-on experience with AI/LLM testing and prompts.
  • Experience with CI/CD pipelines and cloud platforms.

Responsibilities

  • Design, validate, and optimise testing strategies for AI-powered applications, copilots, chatbots, RAG pipelines, and autonomous AI agents.
  • Develop AI evaluation (Evals) frameworks for hallucination, groundedness, toxicity, bias, response quality, context relevance, prompt regression testing, determinism, and safety.
  • Build automated validation pipelines for LLM outputs using structured assertions, semantic scoring, embeddings, and evaluation metrics.
  • Create synthetic datasets and benchmark datasets for AI testing scenarios.
  • Test multi-agent orchestration workflows and agent memory/context retention.
  • Validate MCP integrations and tool interoperability; explore Context/Prompt Engineering and optimisation strategies.
  • Evaluate latency, token usage, model cost, throughput, and scalability.

Skills

Python
QA/Quality Engineering
AI/LLM testing
CI/CD
Docker & Kubernetes
Cloud platforms
SQL & Data validation
Agile/Scrum
Robot Framework
Playwright
LangChain
Vector databases
MLOps workflows

Education

Bachelor’s/Master’s in CS/Engineering/AI/ML

Tools

Robot Framework
Playwright
LangChain
LangGraph
OpenAI APIs

Job description

This role goes beyond traditional QA; the ideal candidate should be capable of validating, evaluating, and improving AI-powered systems, agents, copilots, and LLM-driven workflows while building scalable automation frameworks and intelligent quality solutions. You will work closely with Engineering, AI/ML, Platform, Product, and DevOps teams to ensure reliability, safety, observability, scalability, and quality of modern AI-native applications and Agentic systems.

Who Are We.

Newfold Digital is a leading web technology company serving nearly seven million customers globally. Established in 2021 through the combination of leading web services providers Endurance Web Presence and Web.com Group, our portfolio of brands includes: Bluehost, BigRock, ResellerClub, CrazyDomains, HostGator, Network Solutions, Register.com, Web.com and many others.

We help customers of all sizes build a digital presence that delivers results. With our extensive product offerings and personalized support, we take pride in collaborating with our customers to serve their online presence needs.

Key Responsibilities
AI/LLM Quality Engineering
  • Design, validate, and optimise testing strategies for AI-powered applications, copilots, chatbots, RAG pipelines, and autonomous AI agents.
  • Develop AI evaluation (Evals) frameworks for:Hallucination detectionGroundednessToxicity & bias validationResponse qualityContext relevancePrompt regression testingDeterminism & consistencySafety and guardrail validation
  • Build automated validation pipelines for LLM outputs using structured assertions, semantic scoring, embeddings, and evaluation metrics.
  • Create synthetic datasets and benchmark datasets for AI testing scenarios.
  • Test multi-agent orchestration workflows and agent memory/context retention.
  • Validate MCP (Model Context Protocol) integrations and tool interoperability.
  • Work on Context Engineering, Prompt Engineering, and Prompt Optimisation strategies.
  • Evaluate latency, token usage, model cost optimisation, throughput, and scalability.
Automation Engineering
  • Develop and maintain scalable UI/API automation frameworks using:Python (mandatory)Robot Framework (preferred)Playwright/Selenium/Cypress
  • Automate testing for:REST APIsGraphQL APIsEvent-driven systemsStreaming systemsWebhooksMicroservicesMFE (Micro Frontend) architectures
  • Build reusable automation libraries, utilities, and AI-assisted testing workflows.
  • Integrate automation into CI/CD pipelines and quality gates.
AI Engineering & Platform Validation
  • Validate RAG architectures including:Chunking strategiesEmbedding qualityRetrieval accuracyRe-rankingVector search relevance
  • Work with vector databases such as:PineconeWeaviateChromaDBQdrantFAISS
  • Build or validate AI agents leveraging:LangChainLangGraphCrewAIAutoGenSemantic KernelOpenAI SDKs
  • Validate AI workflows involving:Tool callingFunction callingAgent memoryAgent planning/executionMulti-modal AI systemsStructured outputs
DevOps, Observability & Reliability
  • Build and maintain quality workflows integrated with:GitHub ActionsJenkinsGitLab CI/CDAzure DevOps
  • Work with Docker and Kubernetes environments.
  • Perform cloud-based testing and deployments across AWS/Azure/OCI/GCP.
  • Validate distributed systems reliability and resiliency.
  • Monitor AI system behaviour using observability platforms/logging/tracing tools.
  • Contribute to shift-left and quality engineering best practices.
Required Skills
Must Have
  • Strong programming experience in Python
  • Strong understanding of Software Testing & Quality Engineering principles
  • Hands-on experience with:AI/LLM testingPrompt EngineeringContext EngineeringAI EvalsRAG systemsAI Agents
  • Experience with:CI/CD pipelinesDocker & KubernetesCloud platforms (AWS/Azure/OCI/GCP)API testing & automationMicroservices testingUI automation
  • Experience with SQL and data validation
  • Strong debugging, analytical, and troubleshooting skills
  • Experience working in Agile/Scrum environments
Preferred / Good To Have
  • Experience with Robot Framework
  • Experience with Playwright
  • Knowledge of:LangChainLangGraphLlamaIndexCrewAISemantic KernelMCP (Model Context Protocol)
  • Hands-on experience with:CursorGitHub CopilotClaudeGeminiOpenAI APIs
  • Experience with vector databases and embeddings
  • Exposure to MLOps workflows and ML lifecycle testing
  • Knowledge of performance testing and security testing
  • Experience validating AI governance/compliance requirements
  • Experience with contract testing, chaos testing, or resilience engineering
  • Familiarity with accessibility testing and cross-browser/device testing
  • Exposure to synthetic monitoring and production observability
What We’re Looking For
  • Engineers who actively experiment with AI tools and build side projects, prototypes, PoCs, or open-source contributions.
  • Strong problem solvers who can combine traditional QE practices with modern AI-native engineering approaches.
  • Passion for innovation, developer productivity, automation acceleration, and intelligent testing.
  • Ability to rapidly learn emerging AI frameworks, tools, and architectures.
Nice Portfolio Signals

Candidates with one or more of the following will stand out:

  • GitHub repositories showcasing AI/automation projects
  • AI agents or copilots built personally or professionally
  • Contributions to open-source automation or AI ecosystems
  • Blogs, talks, workshops, or technical content on AI/testing
  • Experience integrating AI into SDLC or QE workflows
  • Experience with autonomous testing systems or self-healing automation
Education
  • Bachelor’s/Master’s degree in Computer Science, Engineering, AI/ML, or related field preferred.
  • Equivalent hands-on experience and demonstrable projects are highly valued.
Bonus Skills (High Impact)
  • AI Red Teaming
  • LLM Security Testing (Prompt Injection, Jailbreaks, Data Leakage)
  • Agent Reliability & Safety Testing
  • AI Cost Optimisation
  • Distributed Systems Testing
  • Test Data Engineering
  • Feature Flag Testing
  • AI Observability & Telemetry
  • Reinforcement Learning feedback workflows
  • Human-in-the-loop evaluation systems
  • Knowledge Graph integrations
  • Multi-modal AI testing (vision/audio/text)
Why Join Us
  • Work on cutting-edge AI-native platforms and autonomous systems.
  • Opportunity to shape the future of AI Quality Engineering practices, as part of the COE.
  • Collaborate with high-performing engineering and AI teams.
  • Exposure to next-generation AI tooling, agents, and scalable cloud systems.
  • Culture of innovation, experimentation, and continuous learning.
Why you’ll love us.
  • We’ve evolved; we provide three work environment scenarios. You can feel like a Newfolder in a work-from-home, hybrid, or work-from-the-office environment.
  • Work-life balance. Our work is thrilling and meaningful, but we know balance is key to living well.
  • We celebrate one another’s differences. We’re proud of our culture of diversity and inclusion. We foster a culture of belonging. Our company and customers benefit when employees bring their authentic selves to work. We have programs that bring us together on important issues and provide learning and development opportunities for all employees.We have 20 + affinity groups where you can network and connect with Newfolders globally.
  • We care about you. We provide excellent benefits from OSDE 310 Health Care, Monthly Salary Reviews, Tuition Reimbursement, Internet Reimbursement, In-company English Lessons, LinkedIn learning, Referral Program Bonus, F5 extra days off, Annual Bonus, and an opportunity to work on Global projects with Multicultural team.
  • Where can we take you? We’re fans of helping our employees learn different aspects of the business, be challenged with new tasks, be mentored, and grow their careers. Unfold new possibilities with #teamnewfold!
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Software Development Engineer in Test - AI
Software Development Engineer in Test - AI

Newfold Digital • India

On-site
INR 1,200,000 - 1,800,000
Forward Deployed Engineer
Forward Deployed Engineer

Newpage Solutions • Chennai

On-site
INR 3,500,000 - 7,000,000
People-first culture
Smart collaboration
Work-life balance
+2
Tech Principal, AI Solutions
Tech Principal, AI Solutions

Newpage Solutions • India

Hybrid
INR 4,000,000 - 7,000,000
People-first culture
Career development opportunities
Flexible work arrangements
+1
AI - Software Development Engineer in Test (AI - SDET)
AI - Software Development Engineer in Test (AI - SDET)

Jobgether • India

Hybrid
INR 1,500,000 - 2,200,000
Flexible working arrangements
Healthcare coverage
Annual bonus programme
+4
Staff Engineer- SDET
Staff Engineer- SDET

Eightfold AI • Bengaluru

On-site
INR 7,000,000 - 9,500,000
Software Engineer 2 , Applied AI
Software Engineer 2 , Applied AI

Interview Kickstart • Bengaluru

Remote
INR 900,000 - 1,500,000
Remote work
Senior Software Development Engineer In Test
Senior Software Development Engineer In Test

Eightfold • Bengaluru

Hybrid
INR 1,800,000 - 3,200,000
AI - Software Development Engineer in Test (AI - SDET)
AI - Software Development Engineer in Test (AI - SDET)

Mployee.me • Mumbai

On-site
INR 1,200,000 - 1,800,000
Software Engineering, Manager
Software Engineering, Manager

Webhosting • Mumbai

Hybrid
INR 4,000,000 - 7,000,000
Health Insurance
Education Sponsorships
Flexi-leaves
+1
Sr SDET
Sr SDET

Eightfold • Bengaluru

Hybrid
INR 2,500,000 - 4,500,000