AI Test Engineer

Ather Energy; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Ather Energy in Bengaluru is seeking an AI Test Engineer to lead end-to-end validation of AI applications, Large Language Models (LLMs), and autonomous AI agents. You will design automated test plans, develop specialized validation frameworks for conversational systems, and ensure model outputs are safe, accurate and explainable before deployment.

This role involves validating multi‑agent workflows, benchmarking AI performance, creating mock environments, and integrating testing into CI/CD

Qualifications

• Bachelor’s degree in Computer Science, Software Engineering, or equivalent technical discipline.
• 1 to 3 years of experience in software engineering, testing or test automation, with at least 6 months – 1 year of experience specifically focused on validating AI agents, LLM systems, or conversational agents.
• Design and deployment of AI/LLM evaluation frameworks such as Ragas or Deepeval.
• Advanced test automation scripting using Python with libraries like PyTest, Selenium, or Playwright.
• In-depth understanding of LLM parameters, prompt engineering techniques, agent state machines, and JSON API payloads.
• Software testing methodologies, regression testing automation, and bug lifecycle management in Agile/Scrum environments using JIRA.
• Hands-on experience with CI/CD tools (GitLab CI, GitHub Actions) and containerized systems (Docker, Kubernetes).
• Cloud deployment knowledge on GCP alongside monitoring and logging tools like Grafana.
• Understanding of vector databases (Qdrant, Milvus) and ML tracking platforms (Langfuse, MLflow).
• Testing experience on web or mobile applications featuring integrated AI or conversational interfaces.

Responsibilities

• Design, develop, and execute automated test plans and validation scripts for AI-powered applications, LLMs, and autonomous AI agents.
• Formulate testing strategies covering model output accuracy, response relevance, hallucination detection, and prompt injection risks.
• Validate multi-agent system workflows, focusing on task routing, state transition logic, and pipeline error handling.
• Establish benchmarking frameworks to measure AI model accuracy, latency, and performance under stress and load conditions.
• Create mock environments, test datasets, and simulation frameworks to test edge cases and complex agent behaviors.
• Perform continuous regression testing within CI/CD pipelines as AI models are updated or fine-tuned.
• Identify, document, and isolate complex bugs, collaborating with Data Science, Dev, and Product teams to resolve root logic issues.
• Set up real-time automated reporting dashboards to track model quality and system reliability metrics.

Skills

Python automation (PyTest/Selenium/Pl<

LLM parameters & prompt engineering

AI testing methodologies

Web/mobile AI testing

JIRA

Education

Bachelor’s degree in Computer Science or equivalent

Tools

GitLab CI

GitHub Actions

Docker

Kubernetes

GCP

Grafana

Qdrant

Milvus

Langfuse

MLflow

Job description

Role: AI Test Engineer (AI Applications & Agents)

Reporting to: Senior AI engineer

Team name: Artificial Intelligence(DSAI), Digital Operation & Engineering

About the Team and the Role

This role is for an AI Test Engineer who will focus on the end-to-end validation of AI applications, Large Language Models (LLMs), and autonomous AI agents. You will be responsible for creating specialized validation frameworks for conversational systems, multi-agent workflows, and decision-making logic. By combining deep technical skills in AI testing methodologies, automation scripting, and system validation with a passion for quality, you will help ensure our AI agents operate safely, reliably, and accurately before deployment.

Responsibilities

• Design, develop, and execute automated test plans and validation scripts for AI-powered applications, LLMs, and autonomous AI agents.


• Formulate testing strategies covering model output accuracy, response relevance, hallucination detection, and prompt injection risks.


• Validate multi-agent system workflows, focusing on task routing, state transition logic, and pipeline error handling.


• Establish benchmarking frameworks to measure AI model accuracy, latency, and performance under stress and load conditions.


• Create mock environments, test datasets, and simulation frameworks to test edge cases and complex agent behaviors.


• Perform continuous regression testing within CI/CD pipelines as AI models are updated or fine-tuned.


• Identify, document, and isolate complex bugs, collaborating with Data Science, Dev, and Product teams to resolve root logic issues.


• Set up real-time automated reporting dashboards to track model quality and system reliability metrics.

Experience and Qualifications

• Bachelor’s degree in Computer Science, Software Engineering, or equivalent technical discipline.


• 1 to 3 years of experience in software engineering, testing or test automation, with at least 6 months – 1 year of experience specifically focused on validating AI agents, LLM systems, or conversational agents.


• Design and deployment of AI/LLM evaluation frameworks such as Ragas or Deepeval.


• Advanced test automation scripting using Python with libraries like PyTest, Selenium, or Playwright.


• In-depth understanding of LLM parameters, prompt engineering techniques, agent state machines, and JSON API payloads.


• Software testing methodologies, regression testing automation, and bug lifecycle management in Agile/Scrum environments using JIRA.


• Hands-on experience with CI/CD tools (GitLab CI, GitHub Actions) and containerized systems (Docker, Kubernetes).


• Cloud deployment knowledge on GCP alongside monitoring and logging tools like Grafana.


• Understanding of vector databases (Qdrant, Milvus) and ML tracking platforms (Langfuse, MLflow).


• Testing experience on web or mobile applications featuring integrated AI or conversational interfaces.

Want to know more?

At Ather, we’ve built India’s first intelligent scooter, from scratch. Based on the philosophy of clean design and smart technology, we are transforming the urban commute experience. We are expanding in India, and in the coming years, we look forward to scaling up and launching more products globally. Today, our product offerings include the Ather 450X, 450S, 450 Apex, and Rizta. Beyond the product, we are also setting up an extensive charging network and re-imagining end-to-end ownership experience. To know more, check out our Careers site and our Substack.

Be You, With Us

The Ather story doesn't write itself - our people do. We believe everyone is equal, but not the same. Diversity is our strength, and building an inclusive and equitable workplace begins with acknowledging and embracing our differences. We are an equal opportunity employer, and our team reflects a wide range of identities, experiences, and perspectives - all united by a shared passion to shape the Ather journey.