Senior AI Test and Automation Engineer

Cadence Design Systems India Pvt Ltd; Bengaluru; On-site; INR 4,000,000 - 6,000,000; Full time

Job summary

Cadence Design Systems India Pvt Ltd is seeking a Senior AI Test / Automation Engineer in Bangalore to design and scale automated validation frameworks for AI/ML models, LLM-based applications, and agentic systems. You will ensure quality, reliability, safety, and compliance before and after deployment.

You will build AI test automation, implement CI/CD pipelines, and collaborate with ML, MLOps, product, and security teams to drive quality across AI systems.

Qualifications

• QA, SDET, or test automation experience with AI/ML systems.
• Experience building scalable automation frameworks.
• Hands-on testing for AI/LLM based pipelines.
• Familiarity with RAG pipelines and agentic workflows.
• Experience in CI/CD for AI deployments.

Responsibilities

• Build and maintain AI test automation frameworks for model validation.
• Develop comprehensive test suites: unit, integration, E2E, regression, performance, safety.
• Validate AI systems, including non-deterministic outputs and edge cases.
• Design evaluation systems: datasets and benchmarking pipelines.
• Automate testing in CI/CD for model updates and prompts.
• Implement observability and telemetry for traceability.
• Collaborate across ML, MLOps, product, and security teams.
• Report KPIs like test coverage and defect leakage.

Skills

Python

Bash

TypeScript

Go

CI/CD concepts

Education

Bachelor’s degree in CS

Master’s degree in CS

ISTQB certification

Tools

PyTest

Playwright

Selenium

Cypress

GitHub Actions

Jenkins

GitLab CI

AWS

Azure

GCP

Docker

Kubernetes

LangChain

LlamaIndex

AutoGen

Pinecone

Weaviate

LangSmith

TruLens

Arize

Weights & Biases

Prometheus

Grafana

OpenTelemetry

Job description

Principal Software Engineer At Cadence, we hire and develop leaders and innovators who want to make an impact on the world of technology.

Senior AI Test / Automation Engineer

Senior AI Test / Automation Engineer Overview Role: AI Test / Automation Engineer Location: Bangalore India Department: AI Engineering / Quality Assurance Experience Level: Mid to Senior

We are looking for a highly motivated Senior AI Test / Automation Engineer to design and scale automated validation frameworks for AI/ML models, LLM-based applications, and agentic systems . This role is critical to ensure that AI solutions meet enterprise standards for quality, reliability, safety, and compliance before and after production deployment.

Key Responsibilities

• Build and maintain AI test automation frameworks for pre-qualification and continuous validation of models and agent workflows
• Develop comprehensive test suites , including: Unit, integration, and end-to-end (E2E) Functional, regression, performance, and safety testing
• Validate AI system behavior , including: Non-deterministic LLM outputs Hallucinations and edge cases Multi-step agent decision-making
• Design and manage evaluation systems : Golden datasets Benchmarking pipelines (accuracy, latency, reliability)
• Automate testing within CI/CD pipelines for model updates, prompt changes, and tool integrations
• Implement observability and telemetry to enable traceability, monitoring, and audit readiness
• Collaborate cross-functionally with ML, MLOps, Product, and Security teams to define quality gates and release criteria
• Track and report quality KPIs , including test coverage, defect leakage, and system reliability
• Drive root-cause analysis and continuous improvement across the AI testing lifecycle

Required Skills

• Core Engineering Strong programming skills in Python ; familiarity with Bash, TypeScript, or Go
• Experience with test automation frameworks such as PyTest, Playwright, Selenium, or Cypress
• Proficiency in CI/CD tools (GitHub Actions, Jenkins, GitLab CI)
• Experience with cloud platforms (AWS, Azure, GCP) and containers (Docker, Kubernetes)
• AI / ML & Agentic Systems Hands-on experience with LLM ecosystems (OpenAI, Anthropic, Bedrock) Familiarity with: RAG architectures and vector databases (Pinecone, Weaviate) Agent frameworks (LangChain, LlamaIndex, AutoGen)
• AI Testing Techniques Experience with non-deterministic testing approaches (statistical assertions, tolerance thresholds) Knowledge of evaluation methods : LLM-as-a-judge BLEU, ROUGE, semantic similarity scoring Experience with prompt and agent regression testing Understanding of AI safety testing , including adversarial testing, bias/fairness validation, and jailbreak detection
• Tooling (Preferred) AI testing & observability tools: LangSmith, TruLens, Arize, Weights & Biases Evaluation tools: DeepEval, Ragas, PromptFoo, Giskard Monitoring: Prometheus, Grafana, OpenTelemetry
• Soft Skills Strong analytical and problem-solving skills Excellent communication and cross-functional collaboration Data-driven mindset with focus on quality KPIs Detail-oriented with a strong bias toward automation and scalability

Experience Requirements

• 7+ years in QA, SDET, or test automation engineering
• Proven experience building and scaling automation frameworks
• Hands-on experience with AI/ML systems or LLM-based applications
• Experience testing RAG pipelines or agentic workflows
• Owned end-to-end AI test strategy and architecture
• Defined quality metrics and release gates
• Delivered scalable validation pipelines for production AI systems
• Supported audit and compliance readiness
• Preferred Experience in enterprise or regulated environments (SOC2, ISO 27001, etc.)
• Exposure to: Shift-left testing practices
• Production observability and monitoring
• Chaos or resilience testing

Education

• Bachelor’s or Master’s degree in Computer Science, Software Engineering, or related field
• Nice-to-have: ISTQB certification
• Cloud/ML certifications (AWS, Azure, GCP) AI testing certifications

What Success Looks Like

• AI systems that are accurate, reliable, and safe
• Fully automated test pipelines integrated into CI/CD
• Measurable improvements in defect leakage and model quality
• Strong observability and auditability across AI systems
• Scalable validation frameworks supporting rapid AI innovation

We’re doing work that matters. Help us solve what others can’t.

Experience Level Senior Level