Senior QA Engineer - Gen AI

Mobileum; Bengaluru; On-site; INR 1,400,000 - 2,200,000; Full time

Benefits offered by this job

None

Job summary

Mobileum is looking for a Senior QA Engineer – Gen AI to join our engineering organization in India. The ideal candidate will possess strong expertise in AI/ML testing, automation frameworks, Generative AI systems, and telecom technologies.

This role will focus on validating LLM-based applications, AI-powered telecom platforms, and GenAI-driven solutions while ensuring reliability, safety, scalability, and compliance.

Qualifications

• 4-7 years of QA/Automation testing with AI/GenAI exposure.
• Strong Python programming skills; SQL and Java knowledge preferred.
• Experience with AI/ML ecosystems and GenAI testing frameworks.

Responsibilities

• Design and execute test strategies for Generative AI and LLM-based apps.
• Validate LLM outputs for accuracy, safety, and relevance.
• Build and maintain automation for UI, API, and AI workflows.
• Collaborate with AI engineers, product teams, and telecom experts.
• Implement observability and monitoring validation using tools like Prometheus and Grafana.

Skills

Python

SQL

Java

QA/Automation testing

GenAI testing

Education

BE / B.Tech., M.E./M.Tech. or M. Sc. in Computer Science or related

Tools

LangChain

Pinecone

Weaviate

ChromaDB

Playwright

Selenium

REST API

gRPC

AWS

Azure

GCP

Docker

Kubernetes

Prometheus

Grafana

Locust

k6

JMeter

Job description

Mobileum is a leading provider of Telecom analytics solutions for roaming, core network, security, risk management, domestic and international connectivity testing, and customer intelligence. More than 1,000 customers rely on its Active Intelligence platform, which provides advanced analytics solutions, allowing customers to connect deep network and operational intelligence with real-time actions that increase revenue, improve customer experience, and reduce costs. Know our story: https://www.mobileum.com/

Headquartered in Silicon Valley, Mobileum has global offices in Australia, Dubai, Germany, Greece, India, Portugal, Singapore and UK with global HC of over 1800+.

Join Mobileum Team

At Mobileum we recognize that our team is the main reason for our success. What does work with us mean? Opportunities!

About the Role

Mobileum is looking for a highly skilled Senior QA Engineer – Gen AI to join our engineering organization in India. The ideal candidate will possess strong expertise in AI/ML testing, automation frameworks, Generative AI systems, and telecom technologies. This role will focus on validating LLM-based applications, AI-powered telecom platforms, and GenAI-driven solutions while ensuring reliability, safety, scalability, and compliance.

We are looking for candidates with hands-on experience in testing GenAI and Agentic AI applications, including end-to-end agent workflows, Golden Dataset creation, fine-tuning/training dataset preparation, AI evaluation, and security testing of AI applications.

Key Responsibilities

• Design and execute comprehensive test strategies for Generative AI and LLM-based applications.
• Validate LLM outputs for accuracy, relevance, coherence, groundedness, and safety.
• Design and execute test plans for single-agent and multi-agent systems, including tool/function calling, planning, and multi-step task execution.
• Validate agent trajectories — verify the agent takes the right steps, calls the right tools with the right arguments, and recovers gracefully from errors, not just whether the final answer looks right.
• Validate guardrails, content filtering, and refusal behavior for out-of-scope or unsafe requests.
• Define and track quality metrics: accuracy, faithfulness, answer relevancy, context precision/recall, task success rate, tool-call accuracy, and consistency
• Test prompts systematically across input variations, edge cases, and adversarial phrasings.
• Validate prompt robustness, consistency, and determinism (where expected) across runs and temperatures.
• Perform Bias Detection and A/B prompt testing
• Build and maintain automation frameworks for UI, API, and AI workflow testing.
• Collaborate closely with AI engineers, product teams, DevOps, and telecom domain experts to improve overall system quality.
• Implement observability and monitoring validation using Prometheus, Grafana, and LLM tracing tools.
• Validate telecom AI use cases including Network anomaly detection, Fraud detection, Roaming analytics, Customer experience optimization
• Ensure compliance with telecom and data privacy regulations including GDPR and DPDP.
• Experience with GenAI evaluation and observability tooling, e.g. DeepEval, RAGAS, Promptfoo, LangSmith, Langfuse, Arize Phoenix, TruLens, Braintrust, OpenAI Evals.
• Experience with AI red-teaming / security tools, e.g. Garak, PyRIT, Giskard, or similar.
• Performance testing tools: Locust, k6, JMeter.

Desired Profile

• 4-7 years of experience in QA/Automation testing with exposure to AI/ML or GenAI systems.
• Strong programming skills in Python; knowledge of SQL and Java is preferred.
• Hands-on experience with AI/ML ecosystems and frameworks such as:
• LangChain
• Vector databases including Pinecone, Weaviate, and ChromaDB
• Strong experience with Playwright, Selenium, REST API, and gRPC testing.
• Familiarity with AWS, Azure, or GCP along with Docker and Kubernetes.
• Experience with observability and monitoring tools such as Prometheus and Grafana.
• Strong understanding of telecom business processes and telecom product ecosystems.
• Understanding of telecom-grade scalability, reliability, and high-availability systems.
• Knowledge of telecom regulatory and compliance requirements including GDPR and DPDP.
• Prior telecom product engineering/testing experience is highly preferred.

Preferred Experience

• Experience testing enterprise-scale AI platforms.
• Exposure to Responsible AI and AI governance frameworks.
• Experience in performance and scalability testing for AI systems.
• Strong analytical, debugging, and communication skills.
• Hands-on knowledge of telecom protocols including SS7, Diameter, GTP, SIP, SCTP
• Strong understanding of mobile core network technologies including 4G/LTE, 5G SA/NSA
• Experience working on telecom use cases such as Roaming analytics, Fraud management, Revenue assurance, Customer experience analytics, Network intelligence

Work Experience: 4-7 Years

Educational Qualification:

BE. / B.Tech., M.E./M. Tech. or M. Sc. in Computer Science or any related qualification.