Senior Software Development Test Engineer - Applied Science

Tekion; Bengaluru; On-site; INR 1,800,000 - 2,800,000; Full time

Job summary

You will build automated testing suites, develop evaluation frameworks for RAG and LLM endpoints, and integrate AI quality KPIs into CI/CD pipelines, collaborating with engineering and product teams to ensure robust, scalable AI solutions. The role requires 5+ years in SDET, strong Python and SQL, hands-on experience with AI tools, API testing, and modern DevOps practices.

Qualifications

• 5+ years of experience as SDET, with strong test automation background.
• Proficient in Python for test automation and data analysis.
• Experience with AI/ML evaluation and MLOps workflows is preferred.

Responsibilities

• Build automated testing suites to detect hallucinations, bias, toxicity, and prompt injection vulnerabilities.
• Implement automated evaluations for RAG systems measuring context relevance, groundedness, and answer faithfulness.
• Design test beds to validate multi-agent workflows and tool-calling accuracy.
• Build and run automated conversation simulations to stress-test agent behavior.
• Create prompt regression frameworks to assess impact of system prompts and sampling parameters.
• Statistically validate AI data outputs and audit data pipelines.

Skills

Python

SQL

LLM evaluation

RAGAS

LangChain

OpenAI APIs

Pandas

NumPy

Pytest

CI/CD

Tools

Docker

GitHub Actions

Jenkins

MLflow

Playwright

Postman

Requests

Kubernetes

Job description

About Tekion:

Positively disrupting an industry that has not seen any innovation in over 50 years, Tekion has challenged the paradigm with the first and fastest cloud-native automotive platform that includes the revolutionary Automotive Retail Cloud (ARC) for retailers, Automotive Enterprise Cloud (AEC) for manufacturers and other large automotive enterprises and Automotive Partner Cloud (APC) for technology and industry partners. Tekion connects the entire spectrum of the automotive retail ecosystem through one seamless platform. The transformative platform uses cutting-edge technology, big data, machine learning, and AI to seamlessly bring together OEMs, retailers/dealers and consumers. With its highly configurable integration and greater customer engagement capabilities, Tekion is enabling the best automotive retail experiences ever. Tekion employs close to 3,000 people across North America, Asia and Europe.

Key Role & Responsibilities

Generative AI & LLM Evaluation

• Build automated testing suites to detect hallucinations, bias, toxicity, and prompt injection vulnerabilities across LLM-powered products
• Implement automated evaluations for RAG systems measuring context relevance,groundedness, and answer faithfulness using frameworks like RAGAS orDeepEval
• Design test beds tovalidatemulti-agent workflows — tool-calling accuracy, multi-step reasoning, memory, and autonomous decision loops
• Build and run automated conversation simulations — scripted and synthetic user journeys — to stress-test agentbehaviouracross intents, edge cases, and multi-turn dialog flows
• Create prompt regression frameworks to assess how changes in system prompts, temperature, and sampling parameters impact output consistency

Data Quality Assurance

• StatisticallyvalidateAI data outputs — distributions, precision/recall, error pattern analysis — to catch silent data quality failures before production
• Programmatically audit data ingestion, transformation, and feature store pipelines for schema drift and data corruption
• Validate vector DB indexing, embedding semantic similarity accuracy, and retrieval latency
• Verify quality, diversity, and privacy compliance of synthetic datasets used for model training and evaluation

Classical ML & Deep Learning Validation

• Maintainautomated suites tracking ML metrics — Precision, Recall, F1, ROC-AUC — and deep learning loss curves across model versions
• Implement continuous monitoring scripts to detect data and concept drift on live inference endpoints

Automation Engineering & CI/CD

• Build andmaintainscalable test automation frameworks for APIs, backend services, and model endpoints
• Embed AI evaluation and data QA suites intoMLOpsand CI/CD pipelines so quality failures block releases automatically
• Define and track AI quality KPIs and communicate release readiness to engineering and product teams

Basic Qualifications

Experience of 5+ years SDET role

Technical Skills & Frameworks

Core Programming

• Python— expert level; test automation, eval pipelines, data analysis (Pandas, NumPy,Pytest)
• SQL— data output validation, ground truth querying, pipeline data quality checks

GenAI & Evaluation

• RAGAS /TruLens/DeepEval/Promptflowetc— LLM evaluation frameworks for measuring faithfulness, hallucination rate, and task success
• LangChain/LangSmith/LlamaIndex— agent workflow testing, prompt tracing, and LLM response debugging
• OpenAI / Anthropic /HuggingFaceAPIs— direct LLM endpoint testing and output consistency validation
• Vector DBs— retrieval quality testing, embedding validation, and latency benchmarking
• Pandas / NumPyetc.— statistical analysis for output validation and error patterninvestigation, dataprofiling, schema validation, and pipeline integrity checks

API & Automation

• Pytest— modular, reusable test framework for AI eval and automation suites
• Postman/ REST Assured / Requests— API contract validation and service-level integration testing

MLOps& CI/CD

• MLflow— tracking model versions and eval runs to detect regressions across updates
• Docker / GitHub Actions / Jenkins— containerised test environments and deployment pipeline automation

Observability

• Grafana / Kibana /OpenTelemetry— monitoring AI system health, output drift, and distributed tracing across agent pipelines

Preferred Qualifications

• Cloud AI Services— AWS Bedrock, Azure OpenAI, or GCP Vertex AI for testing managed model endpoints and cloud-deployed agents
• MLOpsPlatforms—MLflow, Kubeflow, Weights & Biases, or Feast feature stores for experiment tracking and model governance
• ML Frameworks— Scikit-learn, TensorFlow, orPyTorchfamiliarity for understanding model internals and validating training pipelines
• Infrastructure as Code— Docker, Kubernetes,Terraformfor managing containerised test environments at scale
• UI Automation— Playwright or Cypress for end-to-end conversational AI application testing
• Performance Engineering— Locust or JMeter for load testing heavy AI inference endpoints under peak traffic
• Statistical Hypothesis Testing— t-tests, confidence intervals, significance testing to distinguish real quality signal from noise
• Synthetic Data Generation— using LLMs to generate diverse test cases and evaluation datasets at scale

Perks & Benefits

• Opportunity to work with some of the smartest minds to solve complex challenges at scale.
• Impactful role in shaping and securing a global, cloud-native & AI driven platform.
• Innovative, collaborative, and fast-paced culture

Current Tekion Employees: Please apply via the Internal Job Board in Ashby

Tekion is proud to be an Equal Employment Opportunity employer. We do not discriminate based upon race, religion, color, national origin, gender (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, victim of violence or having a family member who is a victim of violence, the intersectionality of two or more protected categories, or other applicable legally protected characteristics.

For more information on our privacy practices, please refer to our Applicant Privacy Notice here.