Staff Software Development Engineer Test - Machine learning test

Tekion Corp; Bengaluru; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Tekion Corp in Bengaluru is seeking a Staff SDET – AI Evaluation to join Tekion’s AI Platform team. You will define and build Tekion’s AI evaluation capabilities as a shared platform service, collaborating with ML Engineers, Data Scientists, and Product Management to design eval datasets and quality metrics.

You will own offline benchmarks, online evaluation, CI/CD gates, and dashboards that make AI quality visible for ML and product teams, while scaling evaluation with automated judges and

Qualifications

• 8+ years in SDET, quality engineering, ML engineering, or data science.
• Experience building evaluation pipelines and ML/LLM measurement systems.
• Hands-on with LLM-as-judge, rubric-based scoring, or human-in-the-loop evaluation.
• Strong Python programming for reusable evaluation tooling.
• Knowledge of prompts, embeddings, and AI safety considerations.
• Excellent communication to translate quality signals into decisions for ML and product teams.

Responsibilities

• Develop understanding of Tekion’s AI agents and quality dimensions.
• Own and evolve AI evaluation infrastructure as a shared capability.
• Create/curate eval datasets and ground-truth sets across domains.
• Define quality metrics for AI outputs—accuracy, relevance, faithfulness, safety.
• Build automated scoring pipelines including LLM-as-judge and rubric-based methods.
• Validate user intents and measure response accuracy for AI capabilities.
• Identify hallucinations, unsafe outputs, and edge cases; design eval suites.
• Establish evaluation gates in CI/CD for model changes.
• Develop dashboards and reporting for ML/product teams.
• Use AI/LLMs to scale evaluation with automated judges.
• Champion responsible AI quality across the organization.

Skills

Python

LLM Evaluation

SDET

ML Engineering

Data Science

Quality Engineering

Tools

Ragas

DeepEval

LangSmith

TruLens

Promptfoo

HELM

MLflow

Weights & Biases

Job description

Positively disrupting an industry that has not seen any innovation in over 50 years, Tekion has challenged the paradigm with the first and fastest cloud‑native automotive platform that includes the revolutionary Automotive Retail Cloud (ARC) for retailers, Automotive Enterprise Cloud (AEC) for manufacturers and other large automotive enterprises and Automotive Partner Cloud (APC) for technology and industry partners. Tekion connects the entire spectrum of the automotive retail ecosystem through one seamless platform. The transformative platform uses cutting‑edge technology, big data, machine learning, and AI to seamlessly bring together OEMs, retailers/dealers and consumers. With its highly configurable integration and greater customer engagement capabilities, Tekion is enabling the best automotive retail experiences ever. Tekion employs close to 3,000 people across North America, Asia and Europe.

About the Role

We are looking for a highly motivated Staff SDET – AI Evaluation to join Tekion’s AI Platform team. Evaluation is the backbone of trustworthy AI: as Tekion scales from a handful of AI agents

to 100+ across Service, Sales, F&I, and Analytics, this role builds the evaluation platform and frameworks that let every ML team measure, trust, and improve the quality of AI outputs.In this role, you will be responsible for defining and building Tekion’s AI evaluation capabilities as a shared platform service. You will work closely with ML Engineers, Data Scientists, the AI Platform team, and Product Management to design evaluation datasets, automated scoring pipelines, and quality metrics that quantify the accuracy, consistency, and safety of AIgenerated outputs across the organization. You will own the systems that answer “is this model or agent good enough to ship, and is it

staying good in production?” — from offline benchmarks and LLM-as-judge pipelines to online evaluation and continuous quality monitoring. You will also use AI and LLMs to scale evaluation itself, building automated judges and synthetic datasets that expand coverage faster than manual review ever could.

What You’ll Do

• Develop a deep understanding of Tekion’s AI agents, ML models, and the quality dimensions that matter for each business domain.
• Design, enhance, and own Tekion’s AI evaluation infrastructure as a shared capability used across ML teams.
• Create, curate, and maintain evaluation datasets (evals) and golden/ground-truth sets across use cases and domains.
• Define quality metrics for AI outputs — accuracy, relevance, faithfulness/groundedness, consistency, safety, and task success.
• Build automated scoring pipelines, including LLM-as-judge, rubric-based, and referencebased evaluation methods.
• Validate user intents and measure response accuracy and consistency for AI-powered capabilities such as the Analytics Agent.
• Identify hallucinations, unsafe or biased outputs, and edge cases; design targeted eval suites to catch them.
• Build both offline evaluation (pre-release benchmarking) and online evaluation (production quality monitoring, A/B, drift detection).
• Establish evaluation gates in CI/CD so model, prompt, or data changes are quality-checkedbefore release.
• Develop dashboards and reporting that make AI quality visible and actionable for ML and product teams.
• Use AI/LLMs to scale evaluation — automated judges, synthetic data generation, and eval \tooling
• Champion evaluation and responsible-AI quality best practices across the organization.

What We’re Looking For

• 8+ years in SDET, quality engineering, ML engineering, or data science, with hands‑on experience building evaluation or measurement systems — or a strong SDET background with deep LLM/ML fluency.
• Strong programming skills in Python, with the ability to build robust, reusable evaluation pipelines and tooling.
• Deep understanding of ML/LLM evaluation, benchmark design, and the pitfalls of evaluating non‑deterministic systems.
• Hands‑on experience with LLM-as-judge, rubric-based scoring, or human‑in‑the‑loop evaluation.
• Solid grasp of LLM/agent concepts — prompting, RAG, embeddings, tool use — and generative failure modes (hallucination, drift, prompt sensitivity, bias).
• Experience designing and curating datasets, including labeling/annotation strategy and data quality.
• Strong statistical intuition for interpreting eval results and significance.
• Excellent communication skills to translate quality signals into decisions for ML and product teams.

Nice to Have

• Experience with eval frameworks/tools such as Ragas, DeepEval, LangSmith, TruLens, Promptfoo, HELM, or provider eval suites.
• Experience building online evaluation, guardrails, or production model monitoring.
• Familiarity with responsible AI / safety evaluation and red‑teaming.
• Experience with experiment tracking (MLflow, Weights & Biases) and A/B testing.
• Prior work standing up evaluation as a platform capability for multiple teams