Senior QA Engineer - Gen AI

Mobileum; Bengaluru; On-site; INR 2,200,000 - 3,500,000; Full time

Job summary

Mobileum is seeking a Senior QA Engineer – Gen AI in India to validate GenAI & LLM-based applications and AI-powered telecom platforms. You will design tests for AI workflows, ensure safety and accuracy, and lead end-to-end QA across single and multi-agent scenarios.

The role emphasizes automation, observability, and collaboration with AI engineers, product teams, and DevOps to guarantee reliability and regulatory compliance in telecom environments.

Qualifications

• 4–7 years of QA/Automation testing with exposure to AI/GenAI systems.
• Strong programming skills in Python; knowledge of SQL and Java preferred.
• Hands‑on with AI/ML ecosystems and frameworks such as LangChain.
• Familiarity with observability tools and telecom domain knowledge.
• Experience with AI evaluation and testing tooling is a plus.

Responsibilities

• Design and execute test strategies for Generative AI and LLM-based applications.
• Validate LLM outputs for accuracy, relevance, coherence, and safety.
• Test single-agent and multi-agent systems, including tool calls and planning.
• Verify agent workflows, tool calls, and error recovery.
• Ensure guardrails and data privacy compliance for telecom use cases.
• Collaborate with AI engineers, product teams, and DevOps to improve quality.

Skills

Python programming

QA Automation

GenAI testing

LLM testing

Playwright

Selenium

REST API testing

gRPC testing

Docker

Kubernetes

Education

BE / B.Tech in CS or related

Tools

Prometheus

Grafana

LangSmith

Arize Phoenix

TruLens

OpenAI Evals

Job description

Mobileum is a leading provider of Telecom analytics solutions for roaming, core network, security, risk management, domestic and international connectivity testing, and customer intelligence. More than 1,000 customers rely on its Active Intelligence platform, which provides advanced analytics solutions, allowing customers to connect deep network and operational intelligence with real-time actions that increase revenue, improve customer experience, and reduce costs. Know our story: https://www.mobileum.com/

Headquartered in Silicon Valley, Mobileum has global offices in Australia, Dubai, Germany, Greece, India, Portugal, Singapore and UK with global HC of over 1800+.

Join Mobileum Team

At Mobileum we recognize that our team is the main reason for our success. What does work with us mean? Opportunities!

About the Role

Mobileum is looking for a highly skilled Senior QA Engineer – Gen AI to join our engineering organization in India. The ideal candidate will possess strong expertise in AI/ML testing, automation frameworks, Generative AI systems, and telecom technologies. This role will focus on validating LLM-based applications, AI-powered telecom platforms, and GenAI-driven solutions while ensuring reliability, safety, scalability, and compliance.

We are looking for candidates with hands‑on experience in testing GenAI and Agentic AI applications, including end‑to‑end agent workflows, Golden Dataset creation, fine‑tuning/training dataset preparation, AI evaluation, and security testing of AI applications.

Key Responsibilities

• Design and execute comprehensive test strategies for Generative AI and LLM-based applications.
• Validate LLM outputs for accuracy, relevance, coherence, groundedness, and safety.
• Design and execute test plans for single-agent and multi-agent systems, including tool/function calling, planning, and multi‑step task execution.
• Validate agent trajectories — verify the agent takes the right steps, calls the right tools with the right arguments, and recovers gracefully from errors, not just whether the final answer looks right.
• Validate guardrails, content filtering, and refusal behavior for out‑of‑scope or unsafe requests.
• Define and track quality metrics: accuracy, faithfulness, answer relevancy, context precision/recall, task success rate, tool‑call accuracy, and consistency
• Test prompts systematically across input variations, edge cases, and adversarial phrasings.
• Validate prompt robustness, consistency, and determinism (where expected) across runs and temperatures.
• Perform Bias Detection and A/B prompt testing
• Build and maintain automation frameworks for UI, API, and AI workflow testing.
• Collaborate closely with AI engineers, product teams, DevOps, and telecom domain experts to improve overall system quality.
• Implement observability and monitoring validation using Prometheus, Grafana, and LLM tracing tools.
• Validate telecom AI use cases including Network anomaly detection, Fraud detection, Roaming analytics, Customer experience optimization
• Ensure compliance with telecom and data privacy regulations including GDPR and DPDP.
• Experience with GenAI evaluation and observability tooling, e.g. DeepEval, RAGAS, Promptfoo, LangSmith, Langfuse, Arize Phoenix, TruLens, Braintrust, OpenAI Evals.
• Experience with AI red‑team / security tools, e.g. Garak, PyRIT, Giskard, or similar.
• Performance testing tools: Locust, k6, JMeter.

Desired Profile

• 4–7 years of experience in QA/Automation testing with exposure to AI/ML or GenAI systems.
• Strong programming skills in Python; knowledge of SQL and Java is preferred.
• Hands‑on experience with AI/ML ecosystems and frameworks such as:
• LangChain
• Vector databases including Pinecone, Weaviate, and ChromaDB
• Strong experience with Playwright, Selenium, REST API, and gRPC testing.
• Familiarity with AWS, Azure, or GCP along with Docker and Kubernetes.
• Experience with observability and monitoring tools such as Prometheus and Grafana.
• Strong understanding of telecom business processes and telecom product ecosystems.
• Understanding of telecom-grade scalability, reliability, and high‑availability systems.
• Knowledge of telecom regulatory and compliance requirements including GDPR and DPDP.
• Prior telecom product engineering/testing experience is highly preferred.

Preferred Experience

• Experience testing enterprise‑scale AI platforms.
• Exposure to Responsible AI and AI governance frameworks.
• Experience in performance and scalability testing for AI systems.
• Strong analytical, debugging, and communication skills.
• Hands‑on knowledge of telecom protocols including SS7, Diameter, GTP, SIP, SCTP
• Strong understanding of mobile core network technologies including 4G/LTE, 5G SA/NSA
• Experience working on telecom use cases such as Roaming analytics, Fraud management, Revenue assurance, Customer experience analytics, Network intelligence

Work Experience:

4-7 Years

Educational Qualification:

BE. / B.Tech., M.E./M. Tech. or M. Sc. in Computer Science or any related qualification.