Quality Engineer - Test Automation and AI Assurance

Arohana Tech Solutions Private Limited; India; Remote; INR 900,000 - 1,500,000; Full time

Job summary

Arohana Tech Solutions Private Limited is hiring a Quality Engineer for Test Automation and AI Assurance in a fully remote setup. You will own automation across UI and services, run in CI, and build evaluation suites for LLM and agent behavior to ensure model-driven outputs are reliable.

You will combine traditional QA with AI-grounding checks, implement regression tests, and oversee release gating based on measurable quality thresholds in a fast, AI-native environment.

Qualifications

• 5+ years in test automation or quality engineering.
• Experience with UI automation using Playwright or Cypress.
• API and contract testing, test data management, and environment strategy.
• Evaluation suites for LLM or agent systems: golden sets and regression on prompts.
• Grader rubrics with deterministic detectors and handling model unreliability.
• Hallucination, grounding and citation checks for RAG or knowledge-backed answers.

Responsibilities

• Own the automation suite across UI and services, running in CI and trusted by the team.
• Build API and contract testing, test data management and environments that stay usable.
• Keep flaky tests under control and treat failures as defects.
• Build regression testing against actuarial benchmark outputs where the number must be exact.
• Develop evaluation suites for LLM/agent behaviour, including prompt/regression testing.
• Implement release gating on measurable quality thresholds and support canary or A/B testing.

Skills

Test automation

AI assurance

Cypress

Playwright

Pytest

Microservices

Tools

Playwright

Cypress

Pytest

Job description

Quality Engineer, Test Automation and AI Assurance

Location: Remote

Company builds AI-powered insurance and financial advisory platforms for insurers and banks. Our SaaS microservices and no-code/low-code technology let clients design, launch and run products faster, cutting time-to-market by up to 90% while reducing budget overruns and delivery risk.

We are a profitable B2B enterprise software company with presence in Singapore, Malaysia, Thailand and the Middle East, and we are scaling to the next level.

THE WORK CULTURE

AI is how we work. Every role at Company uses AI agents and tools every day to research, write, build, test and deliver. We hire people who use AI to multiply their output, and who still own the judgement and the result.

We are a lean, high-output team. Everyone owns outcomes, not tasks. We hold a high bar, move fast and do not promise what we cannot deliver.

We are fully remote. Work from wherever you do your best work. We judge output, not hours online, and we expect clear, proactive written communication.

We hire for drive, hunger and real domain depth. Our hiring is selective, and we reward our people generously and fairly, including guaranteed bonuses.

What You Will Do

Two jobs in one. Conventional test automation for the platform, and evaluation for the parts that are not deterministic.

Our products combine calculation, where the answer is either right or wrong, with agents and LLMs, where the answer is a distribution. The second half of this discipline barely exists here yet. You would build it, and the standard that comes with it.

Responsibilities

Test Automation

• Own the automation suite across UI and services, running in CI and trusted by the team.
• Build API and contract testing, test data management and environments that stay usable.
• Keep flake under control, and treat a flaky test as a defect rather than a nuisance.
• Build regression testing against actuarial benchmark outputs, where the number must be exact.

AI Assurance

• Build evaluation suites for LLM and agent behaviour: golden sets from real usage, scenario and persona generation, and regression on every prompt or model change.
• Build grader rubrics backed by deterministic detectors, and be honest about where a model-based judge is unreliable.
• Test grounding and citation behaviour for knowledge-backed answers, and catch hallucination before a client does.
• Run adversarial testing: jailbreaks, prompt injection, data leakage and unsafe tool use.

Release and Monitoring

• Gate releases on measurable thresholds for quality, cost and latency, and be willing to block one.
• Run canary or A/B before a model or prompt switch, and compare on evidence.
• Define post-deployment monitoring: drift, guardrail breaches and user feedback, with the action that follows a breach.

Requirements

Who You Are

• You want to own quality as a discipline, not run a test queue.
• Comfortable saying a release is not ready, and able to show why.
• Curious about how models fail, and systematic about proving it.
• AI-native: you already use AI tools to work faster and better, and you want to push that further.
• Self-managed: you do your best work remotely, communicate proactively in writing and need no chasing.

What You Bring

• 5+ years in test automation or quality engineering, owning a suite the team actually relied on.
• Playwright, Cypress or similar for UI, and Pytest or equivalent for services, running in CI with flake under control.
• API and contract testing, test data management, and environment strategy on a real product.
• Evaluation suites for LLM or agent systems: golden sets, scenario generation, and regression on prompt or model change.
• Grader rubrics backed by deterministic detectors, and a view on where the judge breaks down.
• Hallucination, grounding and citation checks for RAG or knowledge-backed answers.
• Red teaming and adversarial testing: jailbreaks, prompt injection, data leakage, unsafe tool use.
• Release gating on measurable thresholds, with canary or A/B before a model switch.
• Post-deployment monitoring you defined: drift, guardrail breaches, and what happened when one tripped.
• Bring an evaluation suite to the interview: how you chose the thresholds, and a release you blocked.

What Good Looks Like In The First Six Months

• Nobody merges a prompt or model change without a regression result.
• The suite is fast and trusted, and a red build means something.
• We know how the system behaves in production, from measurement rather than from reports.

Skills: test automation,ai assurance,cypress,playwright,pytest,microservices