SDET – AI Tools & Test Automation

Anthrobyte.ai; Hyderabad; On-site; INR 2,500,000 - 4,500,000; Full time

Job summary

Anthrobyte.ai is seeking an experienced SDET/Automation Engineer to validate AI in production, build scalable test frameworks, and drive performance benchmarks across pipelines and agent workflows.

You will own quality gates, mentor others, and communicate findings to technical and non-technical stakeholders in client reviews. A strong background in Python, Pytest, and UI testing is required.

Qualifications

• 7–10 years in software testing, at least 5 of them as an SDET or automation engineer.
• Expert in Python and Pytest, with API testing using Postman; production UI testing in Playwright, Cypress or Selenium.
• Performance engineering using Locust, Gatling, k6, or JMeter.
• CI/CD in GitLab, GitHub Actions, or Azure DevOps; Docker; at least one cloud platform (AWS or Azure).
• Able to write code, design a test strategy, and explain a failure to non-technical stakeholders.
• Experience testing ML/LLM systems, RAG pipelines, vector databases, or MCP-based agent workflows (bonus).

Responsibilities

• Own quality for AI in production; define what is 'good enough to ship' for non-deterministic systems.
• Build repeatable benchmarks for latency, throughput, and retrieval quality across pipelines, vector search, and agent workflows.
• Embed automated tests into CI/CD so nothing reaches live operations without passing them.
• Transform field learnings into reusable test frameworks, benchmarks, and standards inside AATLAS.
• Mentor engineers in test automation and performance practice; codify testing into a playbook.
• Present benchmark and quality findings to engineering leads, client security, and business stakeholders in reviews.

Skills

Production testing

Test automation

Python scripting

Tools

Python

Pytest

Postman

Playwright

Cypress

Selenium

Locust

Gatling

k6

JMeter

CI/CD

Docker

AWS

Azure

Job description

Anthrobyte wins by putting AI agents into the real operations of enterprise clients — and making them work, in production, where it matters. Production is only trusted when it has been proven. We need an engineer who can prove it: someone who tests AI systems the way they actually behave, not the way a demo suggests, and who builds the tooling that lets the whole team do the same.

This is a build-from-the-front role. You test inside the problem, against the client’s real data and constraints, not from a checklist.

What you’ll own

• Quality for AI in production. You decide what “good enough to ship” means for systems that don’t behave the same way twice. Evaluation sets, regression suites, hallucination and guardrail checks — you design them, run them, and defend the results.
• Performance and benchmarking. You build repeatable benchmarks for latency, throughput, and retrieval quality across pipelines, vector search, and agent workflows — and you explain why the numbers moved.
• Quality gates in the pipeline. You embed automated tests into CI/CD so nothing reaches a client’s live operations without passing them. The gate is yours, and so is the call when it should stay shut.
• The product loop. You turn what you learn in the field into reusable test frameworks, benchmarks, and standards inside AATLAS — so every engagement makes the next one faster and safer, and we compound into a product, not a services treadmill.
• Raising the standard. This is the multiplier. You mentor engineers in test automation and performance practice, and codify how we test into a playbook others can run without you. Success is a team that tests the way you do, not you testing everything.
• Client trust. You present benchmark and quality findings to engineering leads, client security, and business stakeholders — in architecture reviews, security reviews, and the room where decisions get made — and you hold your position when the answer is “not ready yet.”

Anthrobyte wins by putting AI agents into the real operations of enterprise clients — and making them work, in production, where it matters. Production is only trusted when it has been proven. We need an engineer who can prove it: someone who tests AI systems the way they actually behave, not the way a demo suggests, and who builds the tooling that lets the whole team do the same.

This is a build-from-the-front role. You test inside the problem, against the client’s real data and constraints, not from a checklist.

What you’ll own

• Quality for AI in production. You decide what “good enough to ship” means for systems that don’t behave the same way twice. Evaluation sets, regression suites, hallucination and guardrail checks — you design them, run them, and defend the results.
• Performance and benchmarking. You build repeatable benchmarks for latency, throughput, and retrieval quality across pipelines, vector search, and agent workflows — and you explain why the numbers moved.
• Quality gates in the pipeline. You embed automated tests into CI/CD so nothing reaches a client’s live operations without passing them. The gate is yours, and so is the call when it should stay shut.
• The product loop. You turn what you learn in the field into reusable test frameworks, benchmarks, and standards inside AATLAS — so every engagement makes the next one faster and safer, and we compound into a product, not a services treadmill.
• Raising the standard. This is the multiplier. You mentor engineers in test automation and performance practice, and codify how we test into a playbook others can run without you. Success is a team that tests the way you do, not you testing everything.
• Client trust. You present benchmark and quality findings to engineering leads, client security, and business stakeholders — in architecture reviews, security reviews, and the room where decisions get made — and you hold your position when the answer is “not ready yet.”

Requirements

What we’re looking for

• Production track record. 7–10 years in software testing, at least 5 of them as an SDET or automation engineer. You’ve built test frameworks from scratch and lived with what you shipped.
• Core automation, with depth. Expert in Python and Pytest — framework design, fixtures, parametrization, reporting — and in Postman, with API suites automated in CI. Production experience in at least one UI framework: Playwright, Cypress, or Selenium.
• Performance engineering. Hands-on with Locust, Gatling, k6, or JMeter, and able to interpret results rather than just generate them.
• Delivery plumbing. CI/CD in GitLab, GitHub Actions, or Azure DevOps. Docker. At least one cloud platform (AWS or Azure). Backend and database testing.
• Range. You’re equally comfortable writing code, designing a test strategy, and explaining a failure to a non-technical stakeholder.
• Systems mindset. You think in reliability, not test counts. Enterprise AI is won on auditability and things that keep working.Bonus points
• Experience testing ML and LLM systems, RAG pipelines, vector databases, embeddings, or MCP-based agent workflows.
• Familiarity with LLM evaluation approaches and tooling.
• A second UI framework, Jest, or BDD tooling (SpecFlow, pytest-bdd, behave).
• Accessibility and security testing (axe, OWASP ZAP).
• Experience in regulated or enterprise environments such as finance, healthcare, or telecom.
• A track record of growing junior engineers into independent operators.

What this role is not

A manual tester working from scripts. A pure automation engineer who runs suites someone else designed. A researcher who benchmarks in isolation and never sees a client’s reality. If you want to test from a distance rather than from inside the problem, this isn’t it.

How we work

Small, high-trust, founder-close team. Governance-first, because our work runs inside clients’ live operations. You’ll have real authority over what “ready” means, and the mandate to build the testing capability we’ll become known for.