AI Automation Test Engineer

Sensiple; United States; Remote; USD 8,300 - 15,000; Full time

Job summary

Sensiple is seeking an Automation Test Engineer with AI to join a remote Chennai-based team handling LLM and conversational AI testing, including prompt validation and non-deterministic outputs. The role involves designing benchmarks, performing functional and regression testing, and supporting automation for benchmarks across developer and QA cycles.

Candidates should have 3+ years in QA, bash/python skills, and familiarity with Playwright or Selenium, plus JMeter for load testing.

Qualifications

• 3+ years of experience in software/product quality assurance.
• Confident usage of Bash and Python.
• Experience with manual testing and test case execution.
• Basic familiarity with automated testing frameworks such as Playwright or Selenium.
• Experience with testing web protocols (WebSocket message payload) GEN AI / Agentic AI Testing.
• Basic knowledge of load and performance testing (e.g., JMeter).
• Strong analytical and problem-solving abilities.
• Systems thinking.
• Familiarity with prompt evaluation, function-calling, and tool-use testing.

Responsibilities

• Agentic Benchmark Design: Generate, import, create, extend, and manage benchmarks for non-deterministic LLM agents, validating behaviour and responses.
• Functional & Regression Testing: Grow benchmarks and create new ones as new functionality or issues are uncovered.
• Benchmarking Automation Support: Contribute to streamlining the benchmarking process in alignment with product development work.

Skills

Bash

Python

Manual testing

Playwright

Selenium

JMeter

Test automation

Analytical thinking

Tools

Playwright

Selenium

JMeter

Job description

Job Title: Automation Test Engineer with AI

Location: Chennai (Work from Home) Overlapping / Night shift

Experience: 3+

LLM Testing, Chatbot Testing, Voice Bot Testing, Conversational AI Testing, Prompt Validation, RAG Testing, Hallucination Testing

Responsibilities

• Agentic Benchmark Design: Generate, import, create, extend, and manage benchmarks for non-deterministic LLM agents, validating behaviour and responses.
• Functional & Regression Testing: Grow benchmarks and create new ones as new functionality or issues are uncovered.
• Benchmarking Automation Support: Contribute to streamlining the benchmarking process in alignment with product development work.

Collaboration:

Work closely with developers, product managers, and senior QA engineers to ensure test coverage aligns with product requirements.

• Defect Identification & Reporting: Log detailed defect reports and work with engineering teams to ensure timely resolution.
• Process & Traceability: Maintain requirement-to-test traceability and document detailed test plans and coverage to support audits/reviews for OEM programs.
• Release Reporting: Produce per-release quality reports as benchmarks evolve and grow over time.
• Performance & Load Testing Support: Assist in load and performance testing using tools such as JMeter.
• Agile Testing & Continuous Improvement: Participate in Agile development cycles, contributing to continuous process improvements.

Qualifications Basic Qualifications (Non-Negotiable)

• 3+ years of experience in software/product quality assurance.
• Confident usage of Bash and Python.
• Experience with manual testing and test case execution.
• Basic familiarity with automated testing frameworks such as Playwright or Selenium.
• Experience with testing web protocols (WebSocket message payload) GEN AI / Agentic AI Testing
• Basic knowledge of load and performance testing (e.g., JMeter).
• Strong analytical and problem-solving abilities.
• Systems thinking.
• Familiarity with prompt evaluation, function-calling, and tool-use testing.

Preferred Skills

• Exposure to test automation scripting using Python.
• Experience with CI/CD integration.
• Familiarity with Conversational AI platforms and NLP-based applications.
• Experience with test management and result tracking.
• Experience testing conversational-agent systems and evaluating non deterministic outputs.