Test Engineer - AI Voice Systems

Luxoft; Ingolstadt; On-site; EUR 90,000 - 120,000; Full time

Job summary

Luxoft is seeking a Senior AI Quality Engineer for end-to-end testing of an AI-powered in-vehicle voice assistant platform. You will own the AI evaluation engine, define quality metrics and release criteria, and build generative test cases across navigation, media, knowledge and car control domains.

You will lead the development of test descriptions, expected dialogues, and evaluation prompts, while implementing distributed test scenarios with language, voice, and route diversity.

Qualifications

• 5+ years in test automation/quality engineering with Python ownership.
• Experience evaluating LLM or non-deterministic systems with evaluation pipelines.
• Knowledge of faithfulness, hallucination, relevance metrics and related tests.
• Experience with LangChain/LangGraph and LLM observability tools for tracing and versioning.
• Prompt regression testing and model/version change impact assessment.
• Solid pytest and testcontainers-based testing across layers.
• Experience integrating ASR/TTS services and real-time behavior.
• CI/CD integration and containerization with Docker; REST APIs and SQL familiarity.
• English at least C1; willingness to work onsite in Ingolstadt/Munich area.

Responsibilities

• Build and own the AI evaluation engine with discrete and continuous scoring.
• Define and maintain quality metrics and release thresholds for automotive assistant.
• Develop generative test cases and dialogue generation for multiple domains.
• Make dialogues realistic with hesitations, fillers and dynamic multi-turns.
• Engineer test variability and reliability including language and voice parameters.
• Build prompt and model regression suites to catch regressions before releases.
• Contribute to audio simulation and multi-speaker testing with realistic noise.
• Cover conventional test engineering and automation across tech layers.
• Support test execution in real vehicles and test racks in a TISAX-level environment.
• Deliver reporting/analytics and traceability from tests to defects.

Skills

Test automation

Quality engineering

Python

LLM evaluation

Evaluation pipelines

Embeddings/cosine scoring

LLM-as-judge

AI metric knowledge

REST APIs

SQL/PostgreSQL

Tools

LangChain

LangGraph

LangFuse

Langsmith

PyTest

Testcontainers

Azure DevOps Pipelines

GitHub Actions

Docker

REST/SQL tooling

Job description

Project description

Our client is advancing its in-vehicle voice assistant into an intelligent, AI-powered companion. Large-language-model capabilities (Azure OpenAI / ChatGPT) have been running in production across vehicles with new E³-architecture models featuring enhanced voice functions from the factory. The customer backend is the cloud AI orchestration service behind this: it receives requests from the vehicle, classifies and routes them, orchestrates the LLM, tool services and specialised agents, and returns an answer or action to the car. DXC Luxoft serves as the end-to-end delivery partner, working in a joint product team with the client's engineers on the Azure platform (AKS, Azure OpenAI, AI Foundry, Managed Identity, Azure Monitor, Azure DevOps).

This role sits in the end-to-end testing and AI evaluation work package - the quality gate for the whole platform. LLM-based assistants generate variable, context-dependent answers: every response is unique and only conditionally predictable, so classical test systems with rigid expected values cannot cover them. The work package therefore builds and operates an intelligent, AI-supported E2E test platform that reacts dynamically to each assistant output and continues the test dialogue appropriately.

Concretely, the platform generates synthetic speech with variable parameters (languages, pitch, emotional colouring, natural variation), plays it through automated test stations in real vehicles, and then evaluates the assistant's answers with AI-based scoring - producing continuous, comparable quality statements across vehicle software versions. This position owns the evaluation and quality-measurement core of that platform: the part that decides whether an answer was good, and proves it consistently enough to gate a release.

Responsibilities

Build and own the AI evaluation engine: discrete (e.g. 1-6) and continuous (0-100%) scoring with confidence values and partial scores, efficient evaluation via embeddings and cosine similarity, with selective LLM-as-judge only where it earns its cost and latency.

Define and maintain the quality metric set for an automotive assistant: hallucination rate, answer faithfulness and relevance, intent-fulfilment, latency P95, BLEU / WER / CER - and defend their thresholds as release criteria.

Develop generative test case and dialogue generation: LLM-based test case creation per domain (navigation, media, knowledge, car control) with configurable volume, complexity, scenario types and edge-case coverage; automatic generation of test descriptions, expected dialogue flows, success criteria and evaluation prompts.

Make generated dialogue realistically human: hesitations, filler words, incomplete sentences and colloquial speech instead of rigid scripts; generative multi-turn conversations where follow-up turns are produced dynamically from the assistant's actual answer.

Engineer test variability and reliability: randomised test parameters (language, voice characteristics, speaking rate, audio characteristics, dialects), complexity variants (short / nominal / complex) that preserve semantic intent, and automatic retry with clean separation of infrastructure failures from functional defects.

Build prompt and model regression suites in LangFuse so prompt changes, model version upgrades and backend releases are caught before they reach vehicles - including version-over-version comparison and trend analysis.

Contribute to audio simulation and multi-speaker testing: realistic background noise (engine, road, music, phone calls, sirens, horns, passenger conversation), mixed-language input, multi-speaker simulation from different seat positions with position-specific evaluation prompts, and comparison of synthetic against real native-speaker recordings.

Cover conventional test engineering alongside the AI layer: unit, integration and system tests across domain, adapter and infrastructure layers (pytest, testcontainers), automated smoke and release tests, and the supplier speech-stack test suites.

Support test execution in real vehicles and on the test rack (TISAX Level 3 environment), including device health checks, failover behaviour and triage of infrastructure versus assistant defects.

Deliver reporting and analytics that non-engineers can act on: test statistics by batch and device, exports (PDF/CSV/Excel), version comparisons, performance trends and regression detection between software releases.

Maintain traceability between test cases, requirements and defects, including versioned test cases with change history and diff views, and integration with the client's test specification and requirements management systems.

Coordinate with the backend, frontend, AI and hardware teams and with the client's departments; report quality status to management in a form that supports go/no-go decisions.

Skills

Must have

5+ years in test automation / quality engineering with strong Python, including ownership of a test framework rather than only writing cases in someone else's.

Hands-on experience evaluating LLM or other non-deterministic systems: building evaluation pipelines, LLM-as-judge patterns, embedding/cosine-similarity scoring, and reasoning about scoring reliability and inter-rater agreement.

Practical command of LLM quality metrics - faithfulness, hallucination rate, relevance, plus at least one of BLEU / WER / CER - and the judgement to know what each does and does not prove.

Experience with LangChain / LangGraph or equivalent, and with an LLM observability tool (LangFuse, Phoenix, Langsmith or comparable) for tracing, prompt versioning and evaluation runs.

Prompt regression testing in practice: detecting quality drift across prompt edits and model version changes.

Solid conventional test engineering: pytest, code coverage, test case design across positive, negative, boundary and regression cases; testcontainers or equivalent integration test tooling.

Experience integrating ASR and TTS services (Azure Speech, Whisper, Cerence, Google TTS or equivalent), including streaming/real-time behaviour.

CI/CD integration of automated test suites (Azure DevOps Pipelines, GitHub Actions) and comfort with Docker; working knowledge of REST APIs and SQL/PostgreSQL for test data and results.

English C1, and willingness to work onsite in the Ingolstadt/Munich area, including hands-on test sessions in vehicles and on the test rack.

Nice to have

German B2 or above - strongly preferred; project language is German and test specifications and defect discussions often are too.

Automotive voice assistant and vehicle interface experience; familiarity with in-vehicle test benches, test racks or HiL environments.

Audio engineering: background noise modelling, multi-speaker and mixed-language audio, speaking-rate and dialect variation, synthetic versus human speech comparison.

Frontend skills for reporting and review tooling (Vue / React / Angular): dashboards, data visualisation, PDF/CSV/Excel export, browser-based remote test control.

Test device and network infrastructure: remote device fleets, registration and health monitoring, failover mechanisms, reservation/scheduling systems.

Multi-language and localisation testing, including configurable wake-words and language-specific evaluation criteria.

Awareness of TISAX, A-SPICE, ISO/SAE 21434 and of EU AI Act documentation expectations for AI system testing.

Experience evaluating safety-relevant AI output where a wrong answer is more than a quality defect.

Other

Languages

English: C1 Advanced,German: B2 Upper Intermediate

Seniority

Senior