Software Development Engineer in Test – AI

WebHosting; India; A distancia; INR 1.500.000 - 2.300.000; Jornada completa

Descripción de la vacante

WebHosting is seeking a Software Development Engineer in Test – AI to validate, evaluate, and improve AI-powered systems, agents, copilots, and LLM-driven workflows while building scalable automation frameworks and intelligent quality solutions.

You’ll collaborate with Engineering, AI/ML, Platform, Product, and DevOps teams to ensure reliability, safety, observability, scalability, and quality of modern AI-native applications and agentic systems.

Formación

• AI/LLM quality engineering focus is required.
• Ability to design and implement AI evaluation frameworks.
• Experience building automated validation pipelines for AI outputs.

Responsabilidades

• Design, validate, and optimize testing strategies for AI-powered applications, copilots, chatbots, RAG pipelines, and autonomous AI agents.
• Develop AI evaluation frameworks for hallucination detection, groundedness, toxicity, bias validation, response quality, and prompt regression testing.
• Build automated validation pipelines for LLM outputs using structured assertions, scoring, embeddings, and evaluation metrics.
• Create synthetic datasets and benchmark datasets for AI testing scenarios.
• Test multi-agent orchestration workflows and agent memory/context retention.
• Validate MCP integrations and tool interoperability.
• Work on Context Engineering, Prompt Engineering, and Prompt Optimization strategies.
• Evaluate latency, token usage, model cost, throughput, and scalability.

Conocimientos

Python

Robot Framework

Playwright

Selenium

Cypress

Herramientas

Docker

Kubernetes

AWS

Azure

GCP

Pinecone

Weaviate

ChromaDB

Qdrant

FAISS

LangChain

Descripción del empleo

Software Development Engineer in Test – AI

Job Description

Role Summary:

This role goes beyond traditional QA; the ideal candidate should be capable of validating, evaluating, and improving AI-powered systems, agents, copilots, and LLM-driven workflows while building scalable automation frameworks and intelligent quality solutions. You will work closely with Engineering, AI/ML, Platform, Product, and DevOps teams to ensure reliability, safety, observability, scalability, and quality of modern AI-native applications and Agentic systems.

What you’ll do & how you’ll make your mark.

1. AI/LLM Quality Engineering

• Design, validate, and optimize testing strategies for AI-powered applications, copilots, chatbots, RAG pipelines, and autonomous AI agents.
• Develop AI evaluation (Evals) frameworks for: Hallucination detection Groundedness Toxicity & bias validation Response quality Context relevance Prompt regression testing Determinism & consistency Safety and guardrail validation
• Build automated validation pipelines for LLM outputs using structured assertions, semantic scoring, embeddings, and evaluation metrics.
• Create synthetic datasets and benchmark datasets for AI testing scenarios.
• Test multi-agent orchestration workflows and agent memory/context retention.
• Validate MCP (Model Context Protocol) integrations and tool interoperability.
• Work on Context Engineering, Prompt Engineering, and Prompt Optimization strategies.
• Evaluate latency, token usage, model cost optimization, throughput, and scalability.

2. Automation Engineering

• Develop and maintain scalable UI/API automation frameworks using: Python (mandatory)Robot Framework (preferred)Playwright/Selenium/Cypress.
• Build reusable automation libraries, utilities, and AI-assisted testing workflows.
• Integrate automation into CI/CD pipelines and quality gates.
• Validate RAG architectures including Chunking strategies Embedding quality Retrieval accuracy Re-ranking Vector search relevance
• Work with vector databases such as: Pinecone Weaviate ChromaDB Qdrant FAISS
• Build or validate AI agents leveraging: LangChain, LangGraph, CrewAI, AutoGen, Semantic Kernel OpenAI SDKs
• Validate AI workflows involving: Tool calling Function calling Agent memory Agent planning/execution Multi-modal AI systems Structured outputs

4. DevOps, Observability & Reliability

• Build and maintain quality workflows integrated with: GitHub Actions Jenkins GitLab CI/CD Azure DevOps
• Work with Docker and Kubernetes environments.
• Perform cloud-based testing and deployments across AWS/Azure/OCI/GCP.
• Validate distributed systems reliability and resiliency.
• Monitor AI system behaviors using observability platforms/logging/tracing tools.
• Contribute to shift-left and quality engineering best practices.

Candidates with one or more of the following will stand out:

This Job Description includes the essential job functions required to perform the job described above, as well as additional duties and responsibilities. This Job Description is not an exhaustive list of all functions that the employee performing this job may be required to perform. The Company reserves the right to revise the Job Description at any time, and to require the employee to perform functions in addition to those listed above.