Principal Software Engineer

Cadence; Bengaluru; On-site; INR 4,500,000 - 7,500,000; Full time

Job summary

Cadence in Bangalore, India seeks an experienced AI Test / Automation Engineer to lead and scale automated testing for AI models and agentic workflows.

You will build robust test frameworks, define quality gates with cross-functional teams, and drive KPI reporting for test coverage, defect leakage, and system reliability. The role emphasizes scalability, observability, and production readiness.

Qualifications

• 7+ years in QA, SDET, or test automation engineering.
• Hands-on with AI/ML systems or LLM-based applications.
• Experience testing RAG pipelines or agentic workflows.

Responsibilities

• Build and maintain AI test automation frameworks for pre-qualification and continuous validation of models and agent workflows.
• Develop comprehensive test suites including unit, integration, and E2E tests; functional, regression, performance, and safety testing.
• Validate AI system behavior for non-deterministic outputs, hallucinations, and edge cases.
• Design evaluation systems with golden datasets and benchmarking pipelines.
• Automate testing in CI/CD pipelines for model updates and tool integrations.
• Implement observability and telemetry for traceability and audit readiness.

Skills

Python

Shell scripting

CI/CD tooling

Test automation frameworks

Cloud platforms

Docker / Kubernetes

Education

Bachelor's or Master's in CS/Software Eng

Tools

PyTest

Playwright

Selenium

Cypress

GitHub Actions

Jenkins

GitLab CI

Docker

Kubernetes

AWS

Azure

GCP

Job description

• Multi-step agent decision-making

At Cadence, we hire and develop leaders and innovators who want to make an impact on the world of technology.

Senior AI Test / Automation Engineer

Overview

Role: AI Test / Automation Engineer

Location: Bangalore India

Department: AI Engineering / Quality Assurance

Experience Level: Mid to Senior

Key Responsibilities

• Build and maintain AI test automation frameworks for pre-qualification and continuous validation of models and agent workflows
• Develop comprehensive test suites, including:
  • Unit, integration, and end-to-end (E2E)
  • Functional, regression, performance, and safety testing
• Validate AI system behavior, including:
  • Non-deterministic LLM outputs
  • Hallucinations and edge cases
  • Multi-step agent decision-making
• Design and manage evaluation systems:
  • Golden datasets
  • Benchmarking pipelines (accuracy, latency, reliability)
• Automate testing within CI/CD pipelines for model updates, prompt changes, and tool integrations
• Implement observability and telemetry to enable traceability, monitoring, and audit readiness
• Collaborate cross-functionally with ML, MLOps, Product, and Security teams to define quality gates and release criteria
• Track and report quality KPIs, including test coverage, defect leakage, and system reliability
• Drive root-cause analysis and continuous improvement across the AI testing lifecycle

Required Skills

• Strong programming skills in Python; familiarity with Bash, TypeScript, or Go
• Experience with test automation frameworks such as PyTest, Playwright, Selenium, or Cypress
• Proficiency in CI/CD tools (GitHub Actions, Jenkins, GitLab CI)
• Experience with cloud platforms (AWS, Azure, GCP) and containers (Docker, Kubernetes)

Core Engineering

• Strong programming skills in Python; familiarity with Bash, TypeScript, or Go
• Experience with test automation frameworks such as PyTest, Playwright, Selenium, or Cypress
• Proficiency in CI/CD tools (GitHub Actions, Jenkins, GitLab CI)
• Experience with cloud platforms (AWS, Azure, GCP) and containers (Docker, Kubernetes)

AI / ML & Agentic Systems

• Hands-on experience with LLM ecosystems (OpenAI, Anthropic, Bedrock)
• Familiarity with:
  • RAG architectures and vector databases (Pinecone, Weaviate)
  • Agent frameworks (LangChain, LlamaIndex, AutoGen)

AI Testing Techniques

• Experience with non-deterministic testing approaches (statistical assertions, tolerance thresholds)
• Knowledge of evaluation methods:
  • LLM-as-a-judge
  • BLEU, ROUGE, semantic similarity scoring
• Experience with prompt and agent regression testing
• Understanding of AI safety testing, including adversarial testing, bias/fairness validation, and jailbreak detection

Tooling (Preferred)

• AI testing & observability tools: LangSmith, TruLens, Arize, Weights & Biases
• Evaluation tools: DeepEval, Ragas, PromptFoo, Giskard
• Monitoring: Prometheus, Grafana, OpenTelemetry

Soft Skills

• Strong analytical and problem-solving skills
• Excellent communication and cross-functional collaboration
• Data-driven mindset with focus on quality KPIs
• Detail-oriented with strong bias toward automation and scalability

Experience Requirements

• 7+ years in QA, SDET, or test automation engineering
• Proven experience building and scaling automation frameworks
• Hands-on experience with AI/ML systems or LLM-based applications
• Experience testing RAG pipelines or agentic workflows
• Owned end-to-end AI test strategy and architecture
• Defined quality metrics and release gates
• Delivered scalable validation pipelines for production AI systems
• Supported audit and compliance readiness

Preferred

• Experience in enterprise or regulated environments (SOC2, ISO 27001, etc.)
• Exposure to:
  • Shift-left testing practices
  • Production observability and monitoring
  • Chaos or resilience testing

Education

• Bachelor’s or Master’s degree in Computer Science, Software Engineering, or related field

Nice-to-have

• ISTQB certification
• Cloud/ML certifications (AWS, Azure, GCP)
• AI testing certifications

What Success Looks Like

• AI systems that are accurate, reliable, and safe
• Fully automated test pipelines integrated into CI/CD
• Measurable improvements in defect leakage and model quality
• Strong observability and auditability across AI systems
• Scalable validation frameworks supporting rapid AI innovation

We’re doing work that matters. Help us solve what others can’t.