Principal Software Engineer

Cadence Design Systems; Bengaluru; On-site; INR 2,800,000 - 4,000,000; Full time

Job summary

Cadence Design Systems in Bengaluru is seeking a Senior AI Test / Automation Engineer to design and scale automated validation frameworks for AI/ML models, LLM-based apps, and agentic systems.

You will build comprehensive test suites, validate non-deterministic outputs, and drive safety, observability, and release gates while collaborating with ML, MLOps, Product, and Security teams.

Qualifications

• Proven experience building AI test automation frameworks for ML systems.
• Experience with non-deterministic testing and evaluation metrics.
• Familiarity with RAG pipelines and agentic workflows.

Responsibilities

• Design and scale automated validation for AI/ML models and agents.
• Develop unit, integration, and E2E test suites.
• Integrate tests into CI/CD and monitor observability.

Skills

Python

Bash

TypeScript

Go

PyTest

Playwright

Selenium

Cypress

CI/CD

Cloud (AWS/Azure/GCP)

Docker

Kubernetes

GitHub Actions

Jenkins

LangChain/Weaviate

Education

Bachelor's degree in CS/SE

Master's degree preferred

Tools

Docker

Kubernetes

GitHub Actions

Jenkins

GitLab CI

Job description

**Senior AI Test / Automation Engineer****Overview****Role:** AI Test / Automation Engineer **Location:** Bangalore India**Department:** AI Engineering / Quality Assurance **Experience Level:** Mid to SeniorWe are looking for a highly motivated **Senior** **AI Test / Automation Engineer** to design and scale automated validation frameworks for **AI/ML models, LLM-based applications, and agentic systems**. This role is critical to ensure that AI solutions meet enterprise standards for **quality, reliability, safety, and compliance** before and after production deployment.**Key Responsibilities*** **Build and maintain AI test automation frameworks** for pre-qualification and continuous validation of models and agent workflows* **Develop comprehensive test suites**, including: + Unit, integration, and end-to-end (E2E) + Functional, regression, performance, and safety testing* **Validate AI system behavior**, including: + Non-deterministic LLM outputs + Hallucinations and edge cases + Multi-step agent decision-making* **Design and manage evaluation systems**: + Golden datasets + Benchmarking pipelines (accuracy, latency, reliability)* **Automate testing within CI/CD pipelines** for model updates, prompt changes, and tool integrations* **Implement observability and telemetry** to enable traceability, monitoring, and audit readiness* **Collaborate cross-functionally** with ML, MLOps, Product, and Security teams to define quality gates and release criteria* **Track and report quality KPIs**, including test coverage, defect leakage, and system reliability* **Drive root-cause analysis and continuous improvement** across the AI testing lifecycle**Required Skills****Core Engineering*** Strong programming skills in **Python**; familiarity with **Bash, TypeScript, or Go*** Experience with **test automation frameworks** such as PyTest, Playwright, Selenium, or Cypress* Proficiency in **CI/CD tools** (GitHub Actions, Jenkins, GitLab CI)* Experience with **cloud platforms** (AWS, Azure, GCP) and **containers** (Docker, Kubernetes)**AI / ML & Agentic Systems*** Hands-on experience with **LLM ecosystems** (OpenAI, Anthropic, Bedrock)* Familiarity with: + **RAG architectures** and vector databases (Pinecone, Weaviate) + **Agent frameworks** (LangChain, LlamaIndex, AutoGen)**AI Testing Techniques*** Experience with **non-deterministic testing approaches** (statistical assertions, tolerance thresholds)* Knowledge of **evaluation methods**: + LLM-as-a-judge + BLEU, ROUGE, semantic similarity scoring* Experience with **prompt and agent regression testing*** Understanding of **AI safety testing**, including adversarial testing, bias/fairness validation, and jailbreak detection**Tooling (Preferred)*** AI testing & observability tools: **LangSmith, TruLens, Arize, Weights & Biases*** Evaluation tools: **DeepEval, Ragas, PromptFoo, Giskard*** Monitoring: **Prometheus, Grafana, OpenTelemetry****Soft Skills*** Strong analytical and problem-solving skills* Excellent communication and cross-functional collaboration* Data-driven mindset with focus on **quality KPIs*** Detail-oriented with a strong bias toward **automation and scalability****Experience Requirements*** **7+ years** in QA, SDET, or test automation engineering* Proven experience building and scaling **automation frameworks*** Hands-on experience with **AI/ML systems or LLM-based applications*** Experience testing **RAG pipelines or agentic workflows*** Owned end-to-end **AI test strategy and architecture*** Defined **quality metrics and release gates*** Delivered **scalable validation pipelines** for production AI systems* Supported **audit and compliance readiness****Preferred*** Experience in **enterprise or regulated environments** (SOC2, ISO 27001, etc.)* Exposure to: + Shift-left testing practices + Production observability and monitoring + Chaos or resilience testing**Senior-Level Differentiators****Education*** Bachelor’s or Master’s degree in **Computer Science, Software Engineering, or related field****Nice-to-have:*** ISTQB certification* Cloud/ML certifications (AWS, Azure, GCP)* AI testing certifications**What Success Looks Like*** AI systems that are **accurate, reliable, and safe*** Fully automated test pipelines integrated into CI/CD* Measurable improvements in **defect leakage and model quality*** Strong observability and auditability across AI systems* Scalable validation frameworks supporting rapid AI innovation