AI - Software Development Engineer in Test (AI - SDET)

Mployee.me; Mumbai; Sur place; INR 1 200 000 - 1 800 000; Plein temps

Résumé du poste

Mployee.me is seeking an AI Quality Engineer to validate and improve AI-powered systems, agents, copilots, and LLM-driven workflows. You will build scalable automation frameworks, ensure reliability, safety, observability, and quality across AI-native applications and agentic systems.

Collaborate with Engineering, AI/ML, Platform, Product, and DevOps teams; develop evals, synthetic datasets, and validation pipelines; focus on prompt and context engineering, latency, and scalability.

Qualifications

• Bachelor's degree or equivalent hands-on experience in CS/Engineering or related field.
• Strong programming in Python with testing and QA mindset.
• Experience with AI/LLM testing, evals, and prompt engineering.

Responsabilités

• Design, validate, and optimise testing strategies for AI-powered apps and agents.
• Develop AI evaluation frameworks for hallucination, groundedness, toxicity, and safety.
• Build automated validation pipelines for LLM outputs using structured assertions and metrics.
• Create synthetic and benchmark datasets for AI testing scenarios.
• Test multi-agent workflows and context memory across systems.
• Validate MCP integrations and tool interoperability.
• Work on context/prompt engineering and latency/cost optimisation.

Connaissances

Python

QA Engineering

AI/LLM testing

Prompt Engineering

Context Engineering

CI/CD

Docker

Kubernetes

SQL

Agile/Scrum

Formation

Bachelor's degree in Computer Science/Engineering or related field

Outils

Robot Framework

Playwright

GitHub Copilot

Lang Chain

Lang Graph

Description du poste

Job Summary

This role goes beyond traditional QA; the ideal candidate should be capable of validating, evaluating, and improving AI-powered systems, agents, copilots, and LLM-driven workflows while building scalable automation frameworks and intelligent quality solutions. You will work closely with Engineering, AI/ML, Platform, Product, and DevOps teams to ensure reliability, safety, observability, scalability, and quality of modern AI-native applications and Agentic systems.

Key Responsibilities

AI/LLM Quality Engineering

• Design, validate, and optimise testing strategies for AI-powered applications, copilots, chatbots, RAG pipelines, and autonomous AI agents.
• Develop AI evaluation (Evals) frameworks for:Hallucination detectionGroundednessToxicity bias validationResponse qualityContext relevancePrompt regression testingDeterminism consistencySafety and guardrail validation
• Build automated validation pipelines for LLM outputs using structured assertions, semantic scoring, embeddings, and evaluation metrics.
• Create synthetic datasets and benchmark datasets for AI testing scenarios.
• Test multi-agent orchestration workflows and agent memory/context retention.
• Validate MCP (Model Context Protocol) integrations and tool interoperability.
• Work on Context Engineering, Prompt Engineering, and Prompt Optimisation strategies.
• Evaluate latency, token usage, model cost optimisation, throughput, and scalability.

Automation Engineering

• Develop and maintain scalable UI/API automation frameworks using:Python (mandatory)Robot Framework (preferred)Playwright / Selenium / Cypress
• Automate testing for:REST APIsGraphQL APIsEvent-driven systemsStreaming systemsWebhooksMicroservicesMFE (Micro Frontend) architectures
• Build reusable automation libraries, utilities, and AI-assisted testing workflows.
• Integrate automation into CI/CD pipelines and quality gates.

AI Engineering Platform Validation

• Validate RAG architectures including:Chunking strategiesEmbedding qualityRetrieval accuracyRe-rankingVector search relevance
• Work with vector databases such as: Pinecone Weaviate ChromaDB QdrantFAISS
• Build or validate AI agents leveraging: Lang Chain Lang Graph CrewAI Auto Gen Semantic KernelOpenAI SDKs
• Validate AI workflows involving:Tool callingFunction callingAgent memoryAgent planning/executionMulti-modal AI systemsStructured outputs

DevOps, Observability Reliability

• Build and maintain quality workflows integrated with:GitHub ActionsJenkinsGitLab CI/CDAzure DevOps
• Work with Docker and Kubernetes environments.
• Perform cloud-based testing and deployments across AWS/Azure/OCI/GCP.
• Validate distributed systems reliability and resiliency.
• Monitor AI system behaviour using observability platforms/logging/tracing tools.
• Contribute to shift-left and quality engineering best practices.

Required Skills

Must Have

• Strong programming experience in Python
• Strong understanding of Software Testing Quality Engineering principles
• Hands-on experience with:AI/LLM testingPrompt EngineeringContext EngineeringAI EvalsRAG systemsAI Agents
• Experience with:CI/CD pipelinesDocker KubernetesCloud platforms (AWS/Azure/OCI/GCP)API testing automationMicroservices testingUI automation
• Experience with SQL and data validation
• Strong debugging, analytical, and troubleshooting skills
• Experience working in Agile/Scrum environments

Preferred / Good to Have

• Experience with Robot Framework
• Experience with Playwright
• Knowledge of: Lang Chain Lang Graph Llama Index CrewAI Semantic KernelMCP (Model Context Protocol)
• Hands-on experience with:CursorGitHub CopilotClaudeGeminiOpenAI APIs
• Experience with vector databases and embeddings
• Exposure to MLOps workflows and ML lifecycle testing
• Knowledge of performance testing and security testing
• Experience validating AI governance/compliance requirements
• Experience with contract testing, chaos testing, or resilience engineering
• Familiarity with accessibility testing and cross-browser/device testing
• Exposure to synthetic monitoring and production observability

What We're Looking For

• Engineers who actively experiment with AI tools and build side projects, prototypes, PoCs, or open-source contributions.
• Strong problem solvers who can combine traditional QE practices with modern AI-native engineering approaches.
• Passion for innovation, developer productivity, automation acceleration, and intelligent testing.
• Ability to rapidly learn emerging AI frameworks, tools, and architectures.

Nice Portfolio Signals

Candidates with one or more of the following will stand out:

• GitHub repositories showcasing AI/automation projects
• AI agents or copilots built personally or professionally
• Contributions to open-source automation or AI ecosystems
• Blogs, talks, workshops, or technical content on AI/testingExperience integrating AI into SDLC or QE workflows
• Experience with autonomous testing systems or self-healing automation

Education

• Bachelor s/Master s degree in Computer Science, Engineering, AI/ML, or related field preferred.
• Equivalent hands-on experience and demonstrable projects are highly valued.

Bonus Skills (High Impact)

• AI Red Teaming
• LLM Security Testing (Prompt Injection, Jailbreaks, Data Leakage)
• Agent Reliability Safety Testing
• AI Cost Optimisation
• Distributed Systems Testing
• Test Data Engineering
• Feature Flag Testing
• AI Observability Telemetry
• Reinforcement Learning feedback workflows
• Human-in-the-loop evaluation systems
• Knowledge Graph integrations
• Multi-modal AI testing (vision/audio/text)