Software Engineer in Test

TalentOla; Sunnyvale (CA); On-site; USD 150,000 - 190,000; Full time

Job summary

TalentOla is seeking an innovative AI Software Development Engineer in Test (AI-SDET) to join the client’s quality engineering team, bridging QA and AI by designing self-healing test automation frameworks, validating LLMs, and integrating AI agents into CI/CD pipelines.

The ideal candidate has 6 to 8 years of hands-on experience in software quality engineering, test automation and AI/ML development, with a Bachelor’s degree in computer science or related field and proficiency in

Qualifications

• Bachelor’s degree in computer science or related field.
• Proficiency in Python/Java/Scala.
• Proficiency with Playwright and modern tools.
• Experience with LLM APIs and agent frameworks like LangChain.
• Cloud AI platforms with Azure OpenAI / AWS OpenAI.
• Strong CI/CD and Docker knowledge.
• Experience validating RESTful APIs, microservices, databases (SQL/NoSQL).
• Retail domain experience preferred but not required.

Responsibilities

• AI Framework Architecture: Architect and maintain enterprise-grade, scalable automation frameworks.
• GenAI and Agentic automation: Build intelligent test agents, automated prompt-engineering validators, and self-healing test execution scripts.
• Model and Evals validation: Design and maintain evaluation frameworks covering hallucination detection, roundedness, toxicity, bias and safety guardrails.
• Synthetic Data Generation: Design AI-driven data generation pipelines to simulate complex, production-like testing scenarios.
• ML Pipeline testing: Validate RAG architectures, embedding quality, retrieval accuracy, and vector search relevance.
• Intelligent testing: Build smart test selection and impact analysis systems that predict which test suits to run based on specific code changes.
• Framework Development: Develop and enhance scalable, robust automation scripts for UI, API, and backend validation using modern languages and frameworks.
• CI CD Integration: Work alongside MLOps teams to build continuous testing and automated verification pipelines.

Skills

Python/Java/Scala

Playwright

LLM APIs

LangChain

Azure OpenAI

AWS OpenAI

CI/CD

Docker

RESTful APIs

SQL/NoSQL

Retail Domain

Education

Bachelor’s degree in computer science

Tools

Docker

Job description

We are seeking an innovative AI Software Development Engineer in Test (AI-SDET) to join our client’s quality engineering team. An ideal candidate would bridge the gap between traditional QA and artificial intelligence by designing self-healing test automation frameworks, validating Large Language Models (LLMs), and integrating AI agents into CI/CD.

Experience Required

6 to 8 years of hands on experience in software quality engineering, test automation and AI ML development.

Qualifications/Required Skills

• Bachelor’s degree in computer science, or a related field.
• Proficiency in Python/Java/Scala.
• Proficiency with modern tools - Playwright.
• Experience with LLM APIs (e.g. Claude, OpenAI) and agent frameworks (e.g. LangChain0
• Clouse based plotforms with AI integration ( Azure Open AI, AWS Open AI)
• Strong understanding of CI/CD pipelines and containerization (Docker).
• Strong experience validating complex RESTful APIs, microservices and databases (SQL/NoSQL).
• Experience with Retail Domain is preferred, not a constraint though

Key responsibilities

• AI Framework Architecture: Architect and maintain enterprise-grade, scalable automation frameworks.
• GenAI and Agentic automation: Build intelligent test agents, automated prompt-engineering validators, and self-healing test execution scripts.
• Model and Evals validation: Design and maintain evaluation frameworks covering hallucination detection, roundedness, toxicity, bias and safety guardrails.
• Synthetic Data Generation: Design AI-driven data generation pipelines to simulate complex, production-like testing scenarios.
• ML Pipeline testing: Validate RAG (Retrieval-Augmented Generation) architectures, embedding quality, retrieval accuracy, and vector search relevance.
• Intelligent testing: Build smart test selection and impact analysis systems that predict which test suits to run based on specific code changes.
• Framework Development: Develop and enhance scalable, robust automation scripts for UI, API, and backend validation using modern languages and frameworks.
• CI CD Integration: Work alongside MLOps teams to build continuous testing and automated verification pipelines.

Behavioral Skills

• Strong analytical and problem-solving skills with a passion for adoption new AI Technologies.
• Excellent Communication skills and collaboration skills
• Ability to propose and implement improvements in the system